Project Titanicarus: Part 5 – Building the Filers or “Welcome to the Pit of Despair”

This part of the project is the one I have the least experience with and the one which I’ve spent the most time trying to find a solution that works the way that I need.

To put it bluntly I don’t know if a solution exists that is capable of doing what I want with the level of simplicity I want. Almost every solution I have found has its own unique set of shortcomings, almost all of those are performance or complexity related.

I have been through several levels of insanity trying to get a viable solution implemented, including a momentary period of complete lunacy in which I planned to write my own solution.

Lets look at what I am looking for in a backend filesystem:

  • Multi-chassis striping (for performance & redundancy)
  • Self healing in the event of failure without admin intervention
  • Able to scale up by adding more storage servers
  • It must perform well with lots of small files
  • It must be fast enough that web applications don’t lag
  • Replication over WAN to multiple datacentres
  • Capable of continuing to function when partitioned (WAN down)
  • POSIX style locking (not mandatory, but ideal)

Continue reading “Project Titanicarus: Part 5 – Building the Filers or “Welcome to the Pit of Despair””

Project Titanicarus: Part 4 – Building Load Balancers

Load Balancer

Zen Load Balancer

I am using Zen Load Balancer as the front end to all of the infrastructure in this project. It is simple, robust, provides a nice air gap between application servers and the big bad world and it also doubles as a choke point that we can apply security policies and traffic monitoring when required.

I chose Zen over better known load balancers as it can handle TCP and UDP and it will also do SSL offloading, freeing up some CPU on the app servers if we need it later on.

Zen Load Balancer will sit in front of every service we allow to be seen by the outside world – HTTP, HTTPS, DNS, SMTP, POP, IMAP. Continue reading “Project Titanicarus: Part 4 – Building Load Balancers”

Project Titanicarus: Part 3 – Building the PFSense Firewall

Firewall

PFSense is a FreeBSD firewall distro that is primarily focussed on delivering a very simple and secure firewall solution. I am using it because I’ve used it before in production environments and it proved to be a really reliable workhorse. Installation is incredibly simple, I followed the installation guide on their wiki here. I have set the box up with 2 network interfaces, one for internet access and one for internal network access. Continue reading “Project Titanicarus: Part 3 – Building the PFSense Firewall”

Project Titanicarus: Part 2 – Building the Servers

Servers

You will remember that each island in the design requires 9 servers. All servers are going to run Ubuntu, with the exception of the load balancers (Zen Loadbalancer Distro) and the firewall which I’ll be using PFSense for as it has a bunch of pre-built bells & whistles and a simple GUI to drive it with.

Here’s a copy of the island design to refresh your mind:

Single Island Design Continue reading “Project Titanicarus: Part 2 – Building the Servers”

Project Titanicarus: Part 1 – Building a better web infrastructure

The generous boys at Simtronic have just given me a bunch of new server capacity to stick my personal web infrastructure on, so I thought I’d have a go at building something really scalable, fault tolerant, easy to maintain and of course wildly over spec’d for what I need :-)

My current web infrastructure is a series of virtual machines all over the place (AWS, Customer/Friends Networks etc), the goal is to build myself a series of self healing “islands” that can operate independently if required or together when everything is operating ok. I hope that this will eventually become an infrastructure blueprint for other ventures I get myself tangled up in.

The name – yup it’s a mouthful, but it means something. Titanicarus was inspired by the recent Clickfrenzy debacle in which a local web hosting provider failed to properly scale their infrastructure for the hammering of a lifetime. The name is a combination of Titanic, the unsinkable ship and Icarus, the man who flew too close to the sun, melting his wings and falling to his death.

Web infrastructure needs to be stable, and able to adapt quickly. I’m trying to build this infrastructure so it can scale up and down quickly, reacting to whatever Icebergs might come our way while maintaining a reasonable cost overhead so we don’t melt our wings.

Continue reading “Project Titanicarus: Part 1 – Building a better web infrastructure”

Mafia, Penguins & Muses – How I started Mobile Mafia

Mobile Mafia

I have read a couple of books over the last 12 months that really tickled my inner entrepreneur.. The books in question are Ready Fire Aim and The 4 Hour Work Week.

In the 4 Hour Work Week, Tim Ferriss maps out a framework in which you can build a business that with only 4 hours direct effort a week can support a pretty fun sounding lifestyle. There were a few things that didn’t 100% sit well with me, but the book left me with a new way to think about building businesses and some new things to try out.

The real trouble started when I read Ready Fire Aim. Michael Masterson addressed head on the things that I didn’t like about the 4 Hour Work Week and provided me with a way to get my head past a few of the things that I didn’t like in Ferriss’ approach.

One lubricated evening in September last year, I decided to build a muse. “Muse” is the term Ferris uses to describe his 4 hour businesses. I was sitting on the lounge and got really annoyed that my iPhone’s battery wasn’t able to last the whole day. This annoyed me to the point of actually doing something about it. I didn’t write a letter like a grumpy old man, I didn’t throw the phone out the window and I didn’t get up and plug it in. I jumped online and started looking for manufacturers who could help me build a solution to the problem that I have since discovered so many other people are affected by.

That night Mobile Mafia (www.mobilemafia.com) was born. Over a period of 6 months I imported and tested dozens of different cases and mobile accessories. When I found one I liked, I’d import a couple of hundred and sell them on eBay to see how they performed when real customers used them and gathered feedback on popularity by how quickly they sold. Some were good, others caused me to lose many nights sleep.

After 6 or 7 months of this, I had settled on a couple of options and decided to have a go at selling my first branded case – The GodCharger (www.godcharger.com) a couple of people have asked me to write up my experiences, over the next month or two I plan on writing a bit more about my experiences getting this project up and running, and there have been a hell of a lot of things I’ve learned :-)

Why I hate school, but love Education

“If you don’t build your dream, someone else will hire you to help build theirs.”

With many students about to start a new school year, a new university semester or even a new job, many students will be asking themselves, “Why?”

Does success in the school system correlate to success in life? Or is the school system simply geared towards fact retention and regurgitation? Find out what true education is.

Internet Security Tip #10: Be on the Front Foot

Internet Security Tip #10: Be on the Front Foot

From the moment you notice the compromise to the time you decide to mitigate and remove the threat, always be on the front foot. Have a plan and work aggressively on implementing it. The longer you wait to investigate or act, the more potential victims or corporate loss you may be incurring.

The best made plans will never fully prepare you for the real thing, but they will put you in a position that is much stronger than someone without a plan. Keep your team sharp, run regular drills and tests to ensure that your plan happens naturally and doesn’t turn into a bunch of arm flailing madness.

The other place where being on the front foot is critical is when dealing with any publicity surrounding internet security incidents. The media loves a hacking story and they do get very serious coverage, wether you want it or not. From experience I can tell you the only way to handle this kind of story is to get your story out first and to position it carefully, not allowing interviewers to drag you off message.

When the NBN Hacker incident happened, I was interviewed on TV, Radio and Print media 37 times in 3 days and had to decline a further 13 due to there only being one of me.  Calls started at 6am on the day after the arrest when the Federal Police made their announcement and kept coming and coming.

If you are in a similar situation to me, you are going to need a dedicated person to manage incoming calls and requests and to keep a log of the interviews as they happen. Like you would when collecting evidence, keep notes on each journalist you speak to, their contact info and any notes on how the interview went.

I can’t explain how important being ready for the influx of attention is and having a simple, memorable message to convey is. If I wasn’t prepared for the questions that were thrown at me, be they negative or positive, the process would have been incredibly traumatic instead of the positive working together story that we were able to tell alongside the AFP. Tell your story, lead the story, don’t allow negative angles to be explored, always redirect negatives to the core message.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.

Internet Security Tip #9: Protect yourself

Internet Security Tip #9: Protect yourself

Audit everything regularly, turn security into an operational practice. Don’t take risks while investigating network compromises, if collecting evidence comes at the risk of antagonising an attacker or of causing data loss, look for another way.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.

Internet Security Tip #8: Call for Help!

Internet Security Tip #8: Call for Help!

Most businesses are afraid to take action because internet security incidents are seen as an embarrassing PR issue. Hushing up or not reporting internet crime does not fix the problem, it perpetuates it.

If the incident involved the modification of data on your network or someone has obtained access illegally, the offender can be charged, convicted and sent to jail. If you have put together a solid incident management plan, you should be able to take high quality evidence to police that will result in a conviction.

Being portrayed in the media as having been instrumental in the conviction of a criminal rather than the hopeless victim of yet another internet hacking event is a great PR opportunity for any business or IT organisation.

Involve the police. The earlier the better. Know who to call before the incident has happened, this should be in your Operational Readiness plan. The Australian Federal Police and CERT Australia have excellent cybercrime teams who are very helpful, have their numbers in your plan documentation.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.