Field notes
Field notes from our reviews
Two patterns we see in almost every environment we walk into, written by the person who leads our Technology Operations Reviews.
Backups run. Restores get skipped.
Ask an IT person whether backups are tested and the answer is almost always yes. Ask when someone last restored a real file, from a real backup, to a place where they could open it, and the room goes quiet. In my experience those two questions get very different answers, and only the second one matters.
The pattern repeats from large operations to small firms. The backup job runs, the job reports success, and nobody checks what it captured. Then an incident happens and the gap surfaces: a setting was wrong, or the job was quietly skipping the thing that mattered. I have seen a backup set that covered everything except the production database. We were lucky that time. Windows Volume Shadow Copy still held a copy and we recovered from that. Luck is not a control.
Restore tests get skipped because they are awkward and cost time. With older on-premise systems they were spot checks at best: pull a set of tapes from Iron Mountain, try to bring specific files back, hope. Development teams get a free test every time they copy production into staging, because that is a backup and a restore. Production systems do not get that for free, so it has to be somebody's job.
The fix is ownership, not more software. One named person is accountable for proving restores work, and that person is hands-on enough to actually do it. If you want to be bold, attach a bonus to being able to restore a randomly chosen file at any point in the year. Put the test on a checklist a manager looks at, not something done off the side of a desk.
The one question I would tell a managing partner to ask: "Who is accountable for proving we can restore, and when did they last do it?" A name and a date is a good answer. "The backups are running" is where our review usually starts.
Will Sheldon, Managing Director, Pine IT
Everywhere I look, there are accounts for people who left.
Not occasionally. Everywhere. It is not negligence. The firm grew, providers changed hands, someone set up a tool with its own login, and offboarding became a checklist nobody owns end to end.
Admin rights are the same story. There are always admin permissions nobody can explain. You can see they exist. Nobody wants to switch them off, because nobody knows what will break. Cloud accounts are no better: IAM users left in place "just in case". There is nothing wrong with a break-glass account. A former employee's login kept alive two years after they left is not a break-glass account. It is a liability with a name on it.
At Amazon, access review was a formal process, the way it is at most organisations with a SOC-style control set. You receive a list: these people have access to this, confirm it is still right. In my role I got so many of those that doing them diligently took real time. That is the point. Recertification is work, and it only happens when it is somebody's job.
What I would do this week, before talking to anyone like us:
- Pull the user list from your identity provider and your top five business systems. Mark every name that no longer works for you.
- List every admin. For each one, write down why. If nobody can, that is your finding.
- Put authentication in one place. SAML or OpenID Connect into every system that supports it, even if you do not go as far as group-based authorization. One switch to turn a person off beats twelve.
This is risk management. People are bad at low-probability, high-impact risk. Think about what actually happens if an old account gets used, then compare that with the hour it takes to check. It is not much work, and it needs doing on a schedule, not once.
Will Sheldon, Managing Director, Pine IT
Start with the review
Start with a 20-minute fit call.
Enter your details, choose a platform, then pick a time. Prefer email? Write to info@pineit.ca or use the contact form.