Project Titanicarus: Part 2 – Building the Servers

Servers

You will remember that each island in the design requires 9 servers. All servers are going to run Ubuntu, with the exception of the load balancers (Zen Loadbalancer Distro) and the firewall which I’ll be using PFSense for as it has a bunch of pre-built bells & whistles and a simple GUI to drive it with.

Here’s a copy of the island design to refresh your mind:

Single Island Design Continue reading “Project Titanicarus: Part 2 – Building the Servers”

Project Titanicarus: Part 1 – Building a better web infrastructure

The generous boys at Simtronic have just given me a bunch of new server capacity to stick my personal web infrastructure on, so I thought I’d have a go at building something really scalable, fault tolerant, easy to maintain and of course wildly over spec’d for what I need :-)

My current web infrastructure is a series of virtual machines all over the place (AWS, Customer/Friends Networks etc), the goal is to build myself a series of self healing “islands” that can operate independently if required or together when everything is operating ok. I hope that this will eventually become an infrastructure blueprint for other ventures I get myself tangled up in.

The name – yup it’s a mouthful, but it means something. Titanicarus was inspired by the recent Clickfrenzy debacle in which a local web hosting provider failed to properly scale their infrastructure for the hammering of a lifetime. The name is a combination of Titanic, the unsinkable ship and Icarus, the man who flew too close to the sun, melting his wings and falling to his death.

Web infrastructure needs to be stable, and able to adapt quickly. I’m trying to build this infrastructure so it can scale up and down quickly, reacting to whatever Icebergs might come our way while maintaining a reasonable cost overhead so we don’t melt our wings.

Continue reading “Project Titanicarus: Part 1 – Building a better web infrastructure”

Mafia, Penguins & Muses – How I started Mobile Mafia

Mobile Mafia

I have read a couple of books over the last 12 months that really tickled my inner entrepreneur.. The books in question are Ready Fire Aim and The 4 Hour Work Week.

In the 4 Hour Work Week, Tim Ferriss maps out a framework in which you can build a business that with only 4 hours direct effort a week can support a pretty fun sounding lifestyle. There were a few things that didn’t 100% sit well with me, but the book left me with a new way to think about building businesses and some new things to try out.

The real trouble started when I read Ready Fire Aim. Michael Masterson addressed head on the things that I didn’t like about the 4 Hour Work Week and provided me with a way to get my head past a few of the things that I didn’t like in Ferriss’ approach.

One lubricated evening in September last year, I decided to build a muse. “Muse” is the term Ferris uses to describe his 4 hour businesses. I was sitting on the lounge and got really annoyed that my iPhone’s battery wasn’t able to last the whole day. This annoyed me to the point of actually doing something about it. I didn’t write a letter like a grumpy old man, I didn’t throw the phone out the window and I didn’t get up and plug it in. I jumped online and started looking for manufacturers who could help me build a solution to the problem that I have since discovered so many other people are affected by.

That night Mobile Mafia (www.mobilemafia.com) was born. Over a period of 6 months I imported and tested dozens of different cases and mobile accessories. When I found one I liked, I’d import a couple of hundred and sell them on eBay to see how they performed when real customers used them and gathered feedback on popularity by how quickly they sold. Some were good, others caused me to lose many nights sleep.

After 6 or 7 months of this, I had settled on a couple of options and decided to have a go at selling my first branded case – The GodCharger (www.godcharger.com) a couple of people have asked me to write up my experiences, over the next month or two I plan on writing a bit more about my experiences getting this project up and running, and there have been a hell of a lot of things I’ve learned :-)

Internet Security Tip #10: Be on the Front Foot

Internet Security Tip #10: Be on the Front Foot

From the moment you notice the compromise to the time you decide to mitigate and remove the threat, always be on the front foot. Have a plan and work aggressively on implementing it. The longer you wait to investigate or act, the more potential victims or corporate loss you may be incurring.

The best made plans will never fully prepare you for the real thing, but they will put you in a position that is much stronger than someone without a plan. Keep your team sharp, run regular drills and tests to ensure that your plan happens naturally and doesn’t turn into a bunch of arm flailing madness.

The other place where being on the front foot is critical is when dealing with any publicity surrounding internet security incidents. The media loves a hacking story and they do get very serious coverage, wether you want it or not. From experience I can tell you the only way to handle this kind of story is to get your story out first and to position it carefully, not allowing interviewers to drag you off message.

When the NBN Hacker incident happened, I was interviewed on TV, Radio and Print media 37 times in 3 days and had to decline a further 13 due to there only being one of me.  Calls started at 6am on the day after the arrest when the Federal Police made their announcement and kept coming and coming.

If you are in a similar situation to me, you are going to need a dedicated person to manage incoming calls and requests and to keep a log of the interviews as they happen. Like you would when collecting evidence, keep notes on each journalist you speak to, their contact info and any notes on how the interview went.

I can’t explain how important being ready for the influx of attention is and having a simple, memorable message to convey is. If I wasn’t prepared for the questions that were thrown at me, be they negative or positive, the process would have been incredibly traumatic instead of the positive working together story that we were able to tell alongside the AFP. Tell your story, lead the story, don’t allow negative angles to be explored, always redirect negatives to the core message.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.

Internet Security Tip #9: Protect yourself

Internet Security Tip #9: Protect yourself

Audit everything regularly, turn security into an operational practice. Don’t take risks while investigating network compromises, if collecting evidence comes at the risk of antagonising an attacker or of causing data loss, look for another way.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.

Internet Security Tip #8: Call for Help!

Internet Security Tip #8: Call for Help!

Most businesses are afraid to take action because internet security incidents are seen as an embarrassing PR issue. Hushing up or not reporting internet crime does not fix the problem, it perpetuates it.

If the incident involved the modification of data on your network or someone has obtained access illegally, the offender can be charged, convicted and sent to jail. If you have put together a solid incident management plan, you should be able to take high quality evidence to police that will result in a conviction.

Being portrayed in the media as having been instrumental in the conviction of a criminal rather than the hopeless victim of yet another internet hacking event is a great PR opportunity for any business or IT organisation.

Involve the police. The earlier the better. Know who to call before the incident has happened, this should be in your Operational Readiness plan. The Australian Federal Police and CERT Australia have excellent cybercrime teams who are very helpful, have their numbers in your plan documentation.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.

Internet Security Tip #7: If you notice something, don’t act

Internet Security Tip #7: If you notice something, don’t act

Most people’s gut reaction to an internet security incident is to immediately restore from backup or to remove a machine that has been compromised. The only way to ensure that you handle the incident completely and achieve a successful outcome is to take an evidentiary approach. This requires significant patience, monitoring and investigation into the incident.

Any action other than recording of facts and careful planning can destroy evidence which may help you prevent disaster to your business and assist police in achieving a conviction.

So what can you do?

  • Take timestamped step by step notes in your day book of everything you do, as if you were describing them to someone who didn’t understand IT.
  • Take one of your backup storage systems offline and keep it offline, sealed and secured away from your production network for the duration of the investigation as an archived snapshot for evidence and in case of disaster. Take notes in your daybook of what you do.
  • If you have the ability and budget, replace it with a brand new (empty) storage system to ensure redundancy is maintained during the investigation. Take notes in your daybook of what you do.
  • Keep the original secondary storage online. Take notes in your daybook of what you do.
  • Make backups of the compromised machine(s), ideally without logging into the server. Take notes in your daybook of what you do.
  • Archive backups of all machines being investigated onto an offline media on a daily basis, more often if justified. Seal them and store them securely. Take notes in your daybook of what you do.
  • Investigate the backup logs for the server in question, see what has changed on it. Take notes in your daybook of what you do.
  • If you have access to the network switching equipment the server is plugged into, setup a mirrored port and take some traffic captures to see if there is any unexpected network traffic coming form or going to the server. Products like xplico are great open source forensic tools to use in this situation. Archive all traffic captures with notes of what they are, when they were taken and how you took them. Take notes in your daybook of what you do.
  • Once you’ve got a basic understanding of what is going on, pull together a war room with your operational team and assess the extent of the issue. Do you need external help to further assess the situation and to help recommend a course of action? Who has appropriate experience? Does your Operational Incident Plan cover this situation? Take notes in your daybook of what you do.
  • If possible sandbox all affected machines off such that they can not affect any unaffected machines but you can continue to collect evidence in a controlled manner. Continue recording evidence. Take notes in your daybook of what you do.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.

Internet Security Tip #6: Keep a day book

Internet Security Tip #6: Keep a day book

During an incident and when investigating potential incidents record EVERYTHING on paper.

Timestamp every note with the time, date and location of whatever you’re noting down. If you are talking to someone, record their name and any identification they provide. Keep note of who is present at any activity you are performing during the investigation. If you remember something during the day, diarise it in the day book. If you’re looking at traffic, record IP addresses and anything else you can in as much detail as you require to remember the details potentially 5 years down the track.

Make sure that your day book is stored in a way that is not able to be electronically manipulated or read. Remember you’re using it to record events that may wind up being used in evidence, if there is any potential for it to be tampered with by the person(s) you’re investigating, your evidence may be thrown out of court.

During an investigation of a confirmed compromise, your day book should be the only place you record notes.  All internal communication should be done without corporate email accounts and not using corporate PABX systems. Behave as if every corporate asset is potentially compromised until such time as you can prove that it isn’t.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.

Internet Security Tip #5: Network Monitoring

Internet Security Tip #5: Network Monitoring

If a tree falls in the woods and nobody is there to hear it, does it make a sound? If you don’t have an appropriate network and security monitoring solution operating 24 x 7 on your network, you are flying blind.

Network monitoring should monitor everything on your network, but only alert on anomalies to avoid operations teams from being overwhelmed with alarms. It is very important to not only monitor ports that you’re expecting servers to answer on, but also ports that you’re not expecting them to answer on. A server which suddenly starts answering on port 3309 that has no business doing so is more important to investigate than one which has stopped answering on an expected port, this is surely a sign of something you need to investigate.

Network monitoring should be done from more than one location. It should provide internal, external and privileged views of your network to ensure that you are completely aware of what is going on.

Monitoring is not just about ICMP packets and testing for open ports, it is also about watching logs and traffic. Security information and event management tools are great weapons that provide a very deep awareness of what is going on inside your network. These tools often provide early warnings of of issues you would otherwise be completely unaware of, allowing you to act before a compromise ever happens. They are not cheap, but there are some open source options out there.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.

Internet Security Tip #4: Backup Logs are Your Friend

Internet Security Tip #4: Backup Logs are Your Friend

Backup logs are the canary in the coal mine. They provide an amazing level of passive situational awareness. If they stop coming in from a server, theres something wrong.  If a critical operating system file is changed, you can see exactly what was changed and you can download a copy of the altered binary before you ever have to login to the potentially compromised box.

Backup logs are one of the best tools in your ongoing operational security process, and yet they are more often than not deleted or ignored entirely. One of the best security investments you can make is to build or buy a backup log parser or security event monitor that alerts you to files which have been modified unexpectedly.

One of the biggest mistakes you can ever make is to miss out on the treasure trove of information that analysing backup logs can provide.

About Me.

I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.

This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.

I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.

If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!

Shoot me an email – david@hooton.org or grab me on Social Media for more information.