As software companies, we think we’re safe by chasing the “security carrot”. In reality, it’s our own tools that are leaving us vulnerable.
July 15th, 2020: The Breach
My co-worker Dan and I were meeting over a FaceTime call. Our discussion items were finished but we were talking about random, off-topic rants that we had on our chest (as our meetings usually go).
Just as we’re about the end the call, I glanced over at my other monitor. Dan was in mid-sentence when I darted through, “Holy crap. What the hell?”
“– What?”, Dan replied.
“Bill Gates just got hacked. Wait… did he? It’s coming from his validated Twitter handle”, I responded. I took a screenshot and sent this over to Dan via instant messaging.
“There’s no way that Bill Gates would tweet this”, I added with a lot of speculation.
Looking at my screen with confusion, I started getting concerned with how fast this tweet was spreading. The numbers of hearts and retweets kept climbing as my screen updated in real-time.
I copied the Bitcoin address and pasted it into Twitter’s search. The entire screen was filled almost the same tweet from various accounts. I then saw Apple’s Twitter account appear and watched many other validated Twitter accounts send out the same thing.
In a serious tone, I said “Dan, there is something very wrong with Twitter. This is not good at all.”
My eyes opened
How could this happen? There’s no way Bill Gates would allow anyone to have access to his account without a strong password policy and multi-factor authentication.
I speculated it could have been some sort of deeper script that took control of the accounts, but then Twitter confirmed the breach was due to social engineering through an internal support tool.
I felt terrible for Twitter. No one ever intends to be in their situation.
The problem became clear when I saw screenshots of Twitter’s support tool being leaked online. I looked at these in shock, thinking to myself, “This looks just like the support tools that we’ve built!”
All of a sudden, everything clicked.
We’re in a perfect storm right now
There is a massive wave of privacy and security moving through the Internet. It’s making the Internet a better and safer environment. Unfortunately, multiple fronts are colliding that’s putting users and software companies at risk, greater than ever before.
GDPR and CCPA have “blowback”
Every human should have control and access to their data. Europe’s GDPR and California’s CCPA took steps to try and mandate companies to respect this. Unfortunately, this created unintentional consequences.
These laws required software companies to build data export tools so customers can get their data in a handy ZIP file, but this opens up a major flaw.
It’s now easier than ever before for someone to run a “File > Export” and walk away with your most private discussions in a few kilobytes.
It’s getting easier to phish & socially engineer people
I hear this all of the time: “Why should I care about protecting my data when I have nothing to hide?”
The reality is the most basic pieces of information about you verify your identity with your most sensitive services. Worst yet, these pieces of information can also be used to manipulate the people closest to you.
We can easily reverse engineer your connections and your company hierarchy by just going to LinkedIn. Our culture almost expects our company structures to be publicly available now.
Even at a personal level, the more data we post on social media and the more data that gets leaked in every security breach, the larger our attack surface becomes.
These attack surfaces are increasing daily.
Our login processes are more complicated
Just putting in a username and password is only meant for services that you do not care about nor trust. If it’s something that you care about, you enable multi-factor authentication.
The problem is, these extra steps lead to extra confusion. On top of the typical forgotten password requests that support teams get, they are also being flooded with support requests from customers who are frustrated with using multi-factor authentication.
Software companies never want unhappy customers. So we create a button for our support teams that says “Disable Multi-Factor” authentication.
This exposes a critical flaw.
Our support tools are ignoring the basic concepts of security
The collision of these three fronts is setting up support staff to become high-value targets. Twitter suffered from this collision that allowed:
- The attackers were able to gain access to Twitter’s internal support tool by using social engineering. This means the attackers were able to “trick” Twitter support team members to give the attackers access using a phone and a convincing story (no crazy hacker skills required)
- Once inside the support tool, the attackers opened the settings page for their targeted Twitter accounts and changed the email on the account to an email that the attackers owned. This is where they could also possibly disable multi-factor authentication without Bill Gates being “present” in the process
- This allowed the attackers to send password reset requests to their own email account and sign in without a second level of validation
- After resetting the password, they were able to login to the targeted Twitter accounts, send Tweets (thankfully only asking for Bitcoins) and export all of the data on ~36 high-profile accounts (including private messages)
That’s why the Twitter breach was so brilliant. No crazy understanding of mathematics or programming was required. The attackers were able to bypass a complex level of access just by knowing the right amount of information.
So simple in fact, the reports are coming out stating the attackers were as young as 17-years old.
Support teams have too much power
We’re giving too much power to support teams because we’re trying to respond to customers as fast as possible. We want this process to be easy, but we’re forgetting about the basic principles of security.
Software companies are very prideful in talking about their fancy data centers, their alphabet soup of security certifications, but at the end of the day — we’re all human.
Social engineering will always be the easiest way to gain entry to company systems. All it takes is one simple mistake.
Worst yet, software companies think they are safe because they feel good about adding certain statements in their privacy policies.
“We ask for consent before accessing your account”
This is absolutely foolish. This statement only signals a verbal policy is in place and, quite frankly, it may label companies as a target. Okay, you may have some form of identity validation process before someone clicks the “disable multi-factor” button, but do you really think an attacker is going to ask questions first?
The system itself must be the one validating the questions asked. We also must have the customer be “present” when making changes to any security credential elements or upon requesting access to an account.
“Our entire staff has multi-factor authentication”
It doesn’t matter. So did Twitter. Attackers were able to bypass this because support teams are imperfect humans beings like the rest of us.
“Our tools have logging & 3rd party auditing”
This is nonsense. Do you think logging and auditing are going to scrub the data leak of the 36+ high-value accounts on Twitter? Absolutely not.
The data of these accounts are now floating on the dark web and there isn’t a single person in this world that can do anything about it.
Logging and auditing are only “reactive” measures. It answers the question of “Oh no, we’ve been hacked. What happened?”
We need our solution to be defensive. Defensive measures make the crime harder to commit in the first place.
How we fix this
To prevent this from happening, we need a proper placement of checks & balances. You can have the friendliest support person, but they will become a target if they have too much power.
We should continue to trust support employees, but we need to verify each action by giving control to the customer.
Properly balanced power will protect:
- The customer’s data
- The employee’s job
- The company’s reputation
Stop building buttons that instantly disable multi-factor!
Having a second method of authentication makes it significantly more difficult for attackers to breach an account. The problem is multi-factor is worthless if support employees can just click a button to disable it.
If a support team member disables multi-factor for a customer, the action should be validated by the customer before the change happens. This can be validated by a customer that enters a code from an SMS fallback number or another type of recovery code.
If you are dealing with very sensitive data, customers should be able to choose how their account can be recovered. Sometimes even SMS fallbacks can be too high risk for high-profile accounts. This is where customers should be able to take responsibility and choose “recover with my recovery codes only”.
The control should always be with the customer.
Always require a second factor when changing account credentials
I see this all of the time. Sign in to any online software and visit “My Account”. You’ll probably notice that if you want to change the password on your account, you need to enter the current password first.
That’s great, but the major flaw is I can change the email without any sort of validation or notification to the original email.
If the account email gets changed to a malicious email, the attacker now controls the account. All notifications and password reset powers are in the hands of the attacker.
Whenever a credential element is changed, we need to validate the change using another credential element.
Support employees should be able to initiate the request, but the customer should be the one that validates and completes the change.
All support employees should require a support code to access customer accounts
This is by far one of the most important steps in balancing power. The customer should never lose control of their data.
All of these fancy authentication methods are worthless if support employees can view customer accounts from their support tools. Attackers would just need to breach a support account and then start taking screenshots of the customer’s data.
This is where we put the access control into the customer’s hands.
Notifications, notifications, notifications
Notice how I added an email notification in the picture above?
Transparency wins. Customers will build trust with companies who send them notifications about who is accessing their data.
The more transparent we can be in this process, the better it will be for everyone involved. It can even prevent or detect breaches for us.
Support staff access should be “read-only” by default
Permissions are best granted on a “need-to-have” basis. Most of the time we just instantly give “read & write” access to support teams. This allows them to see and make changes to customer data.
Instead of defaulting to “read & write” permissions, only allow support staff to view the account. This prevents breached support accounts from making malicious changes on the platform.
If support staff needs “write” access to accounts, then isolate it to the higher tiers only. Highly train these individuals like SAS operatives who can detect social engineering attempts.
Support employees should not be able to export customer data
Even if there is temporary access granted from the customer, block the data export functions.
There isn’t a good reason why anyone would ever need this.
Software companies: This is a lot of work!
It definitely is, but it’s the responsible and right thing to do. At the end of the day, this protects the software company more than anything.
Reputations will be maintained and even improved with this approach.
Most companies will look at this and label it as a “known improvement” but it will never become a priority — until something bad happens.
If these companies are true gamblers, they will leave their support access “as-is”, but they MUST make it clear that employees can access customer data without the customer being “present”.
I don’t mean “clear” by stating something like “we will ask for your consent before proceeding” in the privacy policy. The privacy policy must clearly state “our support employees can access customer data without customers being present”.
If support employees can disable multi-factor authentication without any automated validation process, this should clearly be stated in the privacy policy as well.
Any misleading or public relation strategies to soften this statement should hold 100% liability to the software company in the event of a support tool breach.
I can’t help. I don’t work for a software company
You can still make a huge impact if you’re not employed by a software company. Software companies are driven by customer demand.
Take a moment and write down the most sensitive data that you have online. Who is hosting that data for you? What would the world look like if that data became publicly available?
Start going through each company’s privacy policy on their website.
Look for something along the lines of “If at any point we need to access your account to help you with a support case, we will ask for your consent before proceeding”. This is the flag you are looking for.
Find their support email and send a message like this:
Hey guys,
I am a huge fan of {{insert company name}}. I read your privacy policy, but I am looking for a detailed explanation of how your employees can access my data.
Can your employees access my data without me being "present"?
If you get a bunch of public-relations-speak, ask the question again. Make them clearly understand what you are asking about and they may state something like “Yes it could be possible, but…”
This is where you know your data is vulnerable. The two major scenarios that are threatening your data at this point are:
- Their support tools being compromised by an attacker
- An employee “going rogue” and accessing customer accounts inappropriately
Let me be clear: Support teams are not evil people. They are not trying to steal your data.
The support staff you are working with probably are realizing this is a major issue at the same time you are. Be respectful and work together on a solution. Be friendly and follow up with them on how they plan to solve the problem.
If you need help explaining your concerns, just send them a link to this post.
Let’s have an open discussion
Twitter should not be raked across the coals over this one. They were just doing everything that everyone in the industry was already doing.
As an industry, we need to own this problem and accept we need to change. If it didn’t happen to Twitter, it would have happened to someone else.
We’re still in the wild west of Internet security and privacy. The only way we can make it safer is by having an open discussion and helping each other towards a solution.
Hopefully, we can all learn from a kid who wanted Bitcoins before this turns into something more dramatic. Humanity dodged a bullet. We’re lucky to still have the opportunity to work together and make the Internet a safer and better place.
If you have thoughts or any additional solutions, please drop them down into the Twitter thread below.
What companies have you found that could be vulnerable?
Cover Photo by Philipp Katzenberger on Unsplash
Update: 08-12-2021
Notion saw this posted and fixed the issue! ?
