Most people’s gut reaction to an internet security incident is to immediately restore from backup or to remove a machine that has been compromised. The only way to ensure that you handle the incident completely and achieve a successful outcome is to take an evidentiary approach. This requires significant patience, monitoring and investigation into the incident.
Any action other than recording of facts and careful planning can destroy evidence which may help you prevent disaster to your business and assist police in achieving a conviction.
So what can you do?
- Take timestamped step by step notes in your day book of everything you do, as if you were describing them to someone who didn’t understand IT.
- Take one of your backup storage systems offline and keep it offline, sealed and secured away from your production network for the duration of the investigation as an archived snapshot for evidence and in case of disaster. Take notes in your daybook of what you do.
- If you have the ability and budget, replace it with a brand new (empty) storage system to ensure redundancy is maintained during the investigation. Take notes in your daybook of what you do.
- Keep the original secondary storage online. Take notes in your daybook of what you do.
- Make backups of the compromised machine(s), ideally without logging into the server. Take notes in your daybook of what you do.
- Archive backups of all machines being investigated onto an offline media on a daily basis, more often if justified. Seal them and store them securely. Take notes in your daybook of what you do.
- Investigate the backup logs for the server in question, see what has changed on it. Take notes in your daybook of what you do.
- If you have access to the network switching equipment the server is plugged into, setup a mirrored port and take some traffic captures to see if there is any unexpected network traffic coming form or going to the server. Products like xplico are great open source forensic tools to use in this situation. Archive all traffic captures with notes of what they are, when they were taken and how you took them. Take notes in your daybook of what you do.
- Once you’ve got a basic understanding of what is going on, pull together a war room with your operational team and assess the extent of the issue. Do you need external help to further assess the situation and to help recommend a course of action? Who has appropriate experience? Does your Operational Incident Plan cover this situation? Take notes in your daybook of what you do.
- If possible sandbox all affected machines off such that they can not affect any unaffected machines but you can continue to collect evidence in a controlled manner. Continue recording evidence. Take notes in your daybook of what you do.
I am a telco & internet entrepreneur, nerd wrangler and massive lover of bacon. I was involved in the investigation of and successful conviction of David Noel Cecil – “Evil, The NBN Hacker”.
This article is written from the experiences I had before, during and since the successful execution of Operation Damara. My experiences are from a telco perspective however they are just as applicable to corporate and government networks.
I currently work with several Australian Telco’s and IT businesses, developing Incident Response Plans and helping them ensure the ongoing continuity of their mission critical infrastructure.
If your organisation doesn’t have a strategy for dealing with internet security incidents, I would love to help you out!
Shoot me an email – firstname.lastname@example.org or grab me on Social Media for more information.