City Hall IT

18,000 alerts before coffee: a brute-force attempt on our VPN

A low-and-slow login attack on our GlobalProtect VPN, how we stopped it, what the logs showed, and why we finally blocked inbound traffic from outside the US.

File no.
CH-0006
Filed
Reading time
6 min
On this page
  1. Not a humidity sensor
  2. Why this one got my attention
  3. Stopping it
  4. Making sure nothing got in
  5. Geoblocking, which we should have done sooner
  6. What we changed, and what I'd tell you

It started like a normal Tuesday. I got to the office, caught up with the team about their evenings, and heard the coffee machine beep that the pot was ready. I filled my cup and sat down to go through email.

One of the best things I've ever done is set up a thorough set of Outlook rules, so my inbox only shows what matters. (That probably deserves a post of its own.) I worked through my own mail, then moved on to the shared mailbox where our alerts go.

It had 18,000 new messages.

Not a humidity sensor

We'd had a long stretch of rainy, humid weather, so my first thought was that a temperature and humidity sensor at one of our sites was flapping. That's happened before, though never at this scale. No panic yet.

Then I opened the folder. These weren't sensor alerts. They were from the firewall, all marked critical, and new ones were arriving as fast as I could read them. Every one was an authentication failure on our VPN.

Now there was panic. We run a Palo Alto Networks firewall with the GlobalProtect portal and gateway published to the internet, and someone was trying to brute-force their way in.

Why this one got my attention

Palo Alto had already disclosed CVE-2026-0257, an authentication bypass in GlobalProtect that could let an attacker establish an unauthorized VPN connection, and exploitation attempts had been reported. When the advisory came out we confirmed our configuration wasn't vulnerable, and we upgraded PAN-OS to a fixed release anyway.

If you haven't checked yours: exploitation requires the portal or gateway to be enabled, authentication override cookies to be turned on (not the default), and the override certificate to be reused or exposed, for example the same certificate GlobalProtect uses for HTTPS. Patching is the fix. Palo Alto's advisory lists the fixed versions and workarounds.

So I was fairly confident we weren't exposed to that bug. But 18,000 failed logins is not something you wait out. A quick search didn't turn up reports of a broad campaign, so this looked targeted at whoever happened to be found running GlobalProtect.

Stopping it

The GlobalProtect logs in the firewall console matched the email alerts: repeated sign-in attempts, every one failing on an invalid username or password. Our GlobalProtect sign-in goes through SAML with MFA, so even a correct password wouldn't have been enough on its own. Still, the logins were failing, and I needed to stop the attempts.

We already follow some of Palo Alto's internet gateway best practices, including blocking the built-in external dynamic lists of known malicious, bulletproof hosting and high-risk IP addresses. None of these addresses were on those lists.

So I built my own:

  1. Created an address group of the source IPs showing up in the GlobalProtect logs.
  2. Added an inbound security rule to drop traffic from that group.
  3. Committed, and repeated. The attacker's addresses were changing faster than I could add them, so this took several rounds.

We also turned off the GlobalProtect portal and added a rule blocking traffic to the gateway, so nothing else could get through. With all of that in place, the attempts stopped about an hour after I'd opened that mailbox.

Making sure nothing got in

There was no sign of compromise, but "no sign" isn't the same as "we checked." We exported the logs and went through them in detail, which took most of the day. Nothing turned up. Our best guess is a script kiddie or a small group working through a list of GlobalProtect portals found on the internet.

The logs did show some interesting things:

  • It was low and slow. Each IP address made only a couple of attempts a minute, and only a few attempts in total, spread across many addresses. Per source, it looked like a normal user who'd forgotten their password, so no single address crossed our intrusion prevention thresholds. What caught it was the volume of failed sign-ins overall. Alert on the total, not just per source.
  • It mimicked a real client. The connections presented the correct GlobalProtect client version. The reported operating system was Windows Server 2012, an odd choice for a laptop VPN user.
  • Most of the addresses I blocked by hand were in the US, while about half of all attempts across the full logs came from outside the US. Keep that in mind for the next part.

Geoblocking, which we should have done sooner

We had never geoblocked inbound traffic. That was a mistake: we have no need for inbound connections from outside the US.

I started small, with a security rule called Non-Approved Source Regions. Its source addresses were the countries that appeared in the attack logs, with the action set to drop. Its hit counter immediately started spinning like a cartoon gas pump, just a blur of numbers. The hit count for drops at the GlobalProtect gateway kept climbing for the rest of the day, but there were no more authentication attempts.

The next morning I checked the numbers first. In about 18 hours, that one rule had dropped more than 750,000 connections. Everything on the internet gets scanned, I know, but that volume was still startling. It raised the obvious question: if we don't do business outside the US, why allow anything from outside the US?

Blocking everything except one country

Palo Alto maintains region objects for countries, and you can define custom regions too. We didn't need custom ones, just a change to the rule.

Blocking everything except one region wasn't intuitive to me, and I couldn't find official documentation for it. A community post on geolocation and geoblocking had the answer: the Negate option.

In the rule's Source tab:

  1. Remove the individual countries from Source Address.
  2. Add United States.
  3. Check Negate under the source address list.

Negate flips the match. Instead of "drop traffic from these places," the rule becomes "drop traffic from anywhere that isn't the US." The rest of the rule stays the same: drop, logged, placed above the rules that allow inbound traffic.

Any firewall change at this scale comes with some nervousness. After the commit, I grabbed my phone, turned off Wi-Fi and checked that our public-facing services still loaded over the cellular network. They did.

Before you do the same

A US-only rule blocks a lot of noise, but it isn't a defense on its own. In our own logs, most of the addresses I blocked by hand were in the US. Attackers rent US cloud servers and residential proxies too. Also check for anything that legitimately needs to reach you from outside the country:

  • Staff who travel internationally and need the VPN
  • Vendors whose remote support comes from overseas
  • Cloud services that call in to your servers, such as webhooks, mail relays or certificate validation (some certificate authorities validate from several regions around the world)

Make exceptions for what you find, rather than skipping the rule.

What we changed, and what I'd tell you

  • Alert on failed sign-ins in total, not just per source. Distributed, low-and-slow attempts won't trip per-IP thresholds. Our failed-authentication alerts were what caught this.
  • MFA on the VPN did its job. Even if a password had been guessed, it wouldn't have been enough.
  • Use the threat intelligence lists your firewall vendor provides, but don't assume they'll catch everything. These addresses weren't on any of them.
  • Geoblock inbound traffic if your organization only serves one country. It's one rule, and it will drop a surprising amount of traffic.
  • When a VPN advisory comes out, check your configuration and patch anyway. We were confident we weren't exposed because we'd done both.

The hit counter on that rule is still climbing. Knowing those connections are dropped before they reach anything of ours is a small reassurance, and one more improvement to our security.