Automatically transcribed, so names and technical terms may be misspelled. The audio is the record.

The Colorado Equals Security podcast is your local source for regional security news, local events, and interviews with key individuals in the region. Now here are your hosts, Robb Reck and Alex Wood. Welcome to Colorado Equals Security. This is the newscast for July 24th, 2017, episode 25. 25!

Alex, we are not in Denver. What are we doing in the desert? Robb, it's hot. It's really, really hot. Alex and I are in Las Vegas.

We're here to kick off the Black Hat week. We're gonna be here for a long, long time, right? Not so much. No, not too long. In and out.

Yeah, so although this, you know, they call it what, Hacker Summer Camp in Vegas, goes for a full week, I'm only here for Sunday and Monday as I'm here for the ISSA CISO event. And Alex, you're just here another day or two, right? Yeah, I'm here a little bit longer than that, but yeah, here for the CISO event and then a a little bit more after that. But while we're physically in Vegas, our minds and our hearts are still in Denver, right? Of course, they never leave.

Well, let's go ahead and get back to where our hearts are. In the news this week, BoJo's on Colorado Boulevard is closing after 40 years of delivering mediocre pizza. How do you feel about that?

I think, Robb, you've already stated how you feel about that. You know, BoJo's is good pizza. It's very nostalgic. You know, I've eaten at the one in Idaho Springs many times, a couple times the one on Colorado Boulevard, so I'm not super disappointed at that one closing. But again, end of a legend.

Yeah, it is inconvenient whenever folks come from out of town, they always wanna go to the place that has honey for your pizza crust, and of course that's what BoJo's has brought to our expectations. Yeah, next on the list, Comcast is launching an Internet of Things network in Denver. So, they're bringing a— it's what they're calling MachineQ. It's a sensor network. So, trying to bring a different method of communication to Internet of Things devices.

They list a bunch of different use cases in this article. You know, sort of, I think, a little bit far out there for a number of them, but they're— I think they're trying to expand their reach. This is, I think, a different sort of low-power network as opposed to the traditional IP network that you may know Comcast for.

The first thing obviously that comes to my mind is, you know, I don't think of Comcast and security necessarily in the same breath. So hopefully they're putting some security resources into this MachineQ network. Well, it's very cool. There are only 20 cities that this is going— excuse me, 12 cities this is going to originally, and of course Denver is one of those. Looking forward to seeing where this goes and what kind of functionality it offers.

Next on the list, the Denver City Auditor has found a security issue with the city's 311 system. So, 311 is the non-emergency version of 911. You can call 311 to ask questions without raising the alarm. But there was an issue found where the vendor, who's— it looks like it's Salesforce, who's being used for this, had access to to citizens' PII after they called, including Social Security numbers and some other sensitive information. Yeah, kudos to the Denver Auditors for checking into this and figuring it out and letting the security folks in Denver know so they get it corrected.

I think this, you know, is one of those issues where you gotta make sure that you're doing your third-party due diligence and checking your vendors, making sure they're securing your data in the way that you expect them to. Yeah, it certainly is not easy, so no huge surprise that there might be an issue with vendor risk management there. Yeah, so next, last week we talked about the Cyber Patriot Summer Camp that the Air Force Academy is doing, and Dan Massey, also we talked about him joining Secure-64, and Dan wrote in, he helps coach a middle school CyberPatriot team and wanted to let us know that, you know, CyberPatriot is a great thing. He's very passionate about it, and he wanted to challenge everyone to sort of semi-quoting here, not just complain about cybersecurity workforce gap, but to help fix it. So donate some time and help coach, you know, middle school, high school kids in CyberPatriot or some other version of of cybersecurity training for young kids.

Yeah, we've talked quite a bit about CyberPatriots, CyberGirls, and really I think it's a big part of trying to get the next generation ready for filling that work gap and getting more diversity into the workforce. So appreciate, Dan, your note there, and of course we definitely recommend those of you listening that you take this seriously and help be a part of the solution. So Ping Identity was named Applied Materials Supplier of the Year. This is a, you know, I work at Ping, this is one of the kind of most proud things we've seen. Applied Materials is a very large multinational organization with a huge supplier ecosystem.

Ping was named the number one vendor for the year based on helping solve a complex problem with single sign-on for their mobile apps in their environment. Awesome. Pretty cool stuff. Kudos to Ping Identity. Next on the list, there was a profile this week in Denver Business Journal Alex Kryline of SecureSet.

Alex is someone that we have interviewed on the podcast before, and it was just neat to see him profiled here. It's actually a video profile, so if you go to the link we have in the show notes, you can see the interview with him. We did reach out to Alex for some comment, and while he was— the profile is on him, he was very gracious in mentioning the other folks at SecureSet along with him that are leading that effort, not just him. Yeah, so Dave Odom is another leader of SecureSet. Said, and Alex wanted to make sure we specifically called out Dave and say, you know, it's not just Alex who's leading up that initiative.

So Alex, thanks a lot, and we're really glad to see that, you know, more visibility for security in Denver. Automox— we mentioned Automox last week when they were giving that CVE explanation blog post. We didn't realize Automox just actually came out of stealth mode this week, so since the last time we recorded. So I guess we jumped the gun. We know more than everybody else.

To be fair, you might not want to subscribe, post blog posts while you're in stealth mode. Well, you know, but regardless, congratulations to those guys for officially coming out. They're in Boulder there, and they're really working on patching and having an innovative solution to getting patches applied to your infrastructure. Yeah, exactly. Next on the list, Red Canary put out a blog post this week, and it's more a set of blog posts really.

It's a number of different resources about threat hunting and how you can do threat hunting, different things around threat hunting. So check out the link that we have in the show notes. It'll link you to a number of stories and resources that they have, some on-demand webinars and videos about threat hunting. You know, it is the topic du jour, so check that out and figure out how to do threat hunting yourself. Yeah, that was really neat.

We certainly— we got a note about that collection of resources from Brianne Houck from Red Canary, and actually got several different notes this week from different folks with articles for us to cover. We appreciate that, and we recommend, you know, if you guys see something in the news you'd like to hear about on the show and get our take, send us a note. We'll look at covering it. Yeah, keep them coming. Thanks.

The final news this week— well, second to final— ProtectWise, actually Tom Hagel over at ProtectWise, has released some research on the Winti threat actor group. And we have a post to that in there. But basically, this is a summary of a threat actor group and talking about the kind of attacks they've done. Really interesting stuff. And it's cool to see that this kind of research is coming out of a Denver-headquartered security company.

Yeah, lots of detailed information in there going really deep into this particular threat actor. It's some great research there. Final thing on the list here, we've talked about previously, CTA has their CISO of the Year Award as part of their APEX Awards. So those nominations are still open. So if you are a CISO or if you are someone who knows a good CISO and you want to nominate them, head on out there and nominate them for CISO of the Year.

And I'd say this is an opportunity for us to reinforce this fantastic progress that we made by even getting this award added. Until this year, there had never been a CISO of the Year award included in the APEX Awards. The last thing we want to see is that they put the award out there and they get 2 nominations. So if you guys know someone who you think might deserve the award, please go ahead and nominate them. Let's, uh, let's blow up their inbox with fantastic nominations of people who are really doing good stuff in security in the Denver area.

Yeah, exactly. So let's go into the calendar of events. Just as a reminder, we do have on our website, colorado-security.com, a list of all of the events coming up in the area over the next several months. And of course, on the show, we just go through the next week or two. So first on the list, the National Cybersecurity Center in Colorado Springs is having the CyberPatriot Cyber Camp along with the Air Force Association.

We mentioned that last week, and again It's good stuff. High school students, if they're interested in cybers, yeah, then get them to go to the camp. Yeah, I'm really looking forward to being able to send my kids to that in a few years. So hopefully it keeps going. But that's Monday through Friday of this week.

Next, on Friday of this week, SecureSet is doing their capture the flag event. And as always, the first hour is kind of an introduction to capture the flags and, you know, give you the background to be successful. And then after that is the more senior, experienced, actual Capture the Flag event itself. On the 29th, ISSA Colorado Springs is having their CISSP exam prep course. We've mentioned this previously as well.

This is a very inexpensive CISSP prep course that Colorado Springs ISSA is offering. I believe— is this the first weekend? This is the first. This is the first weekend, so you still have time to get in and get there for all the different— I think it's 5 or 6 weeks that they do this. Yeah, I think it's 5 weeks.

And I think if you're a member of ISSA Colorado Springs, it's $120 for all 5 weeks. If you're a member of the ISSA Denver chapter, it's $210. So it's really affordable. And if you're not a member of either, I highly recommend you go join one and get on top of this. And then the last thing on the list this week is actually for the next week on July 31st through the 8th.

CTA is doing their tech tour. And on this tech tour, they really drive around the state and do something of an exhibition, getting to know the different cities and talking about what's going on in technology in those areas. There's different dates for the different stops, but they're going to Colorado Springs, Pueblo, Durango, Montrose, Gunnison, Boulder, really a lot of the maybe not so metropolitan areas in Denver, but that are really showing some interesting action on the tech side. Yeah, for sure. And it's not just your normal stops that you would think about.

Maybe you're your Boulder, Denver, Colorado Springs, Fort Collins maybe. Yeah, so good to see that they're going other places as well. All right, let's dive into jobs for sure. So first on the list, Arrow Electronics, they're looking for a principal security architect. Amazon is hiring a senior security engineer.

Cool to see Amazon hiring here for security. LogRhythm is looking for a senior security research engineer, and you might get to work with James Carder and Greg Foss, who are Not terrible people. Definitely not terrible people. Xcel Energy is hiring a senior analyst of threat intelligence. TIAA, no longer TIAA-CREF, is hiring a senior director for IT audit.

The National Renewable Energy Laboratory, NREL, is hiring a cloud computing site reliability engineer. Very cool. Yeah, and one of the focuses of that job is on security, so So it's definitely cool. Is cloud computing involved somehow with solar power? Is that what's happening here?

Clouds get in the way. That's right. I should have stopped this joke a long time ago. Oppenheimer Funds is looking for a security engineer. And Frontier Airlines is hiring a senior network security engineer.

RIM Technologies, which is not BlackBerry, is looking for a data security specialist. But will they let you use a BlackBerry on the job? Potentially. It's a government contract, so most likely you'll be using a BlackBerry. Final job for the week, Ping Identity is hiring.

We're actually looking at changing the name from just GRC analyst to security compliance GRC analyst. Really looking for someone who understands security controls, ISO frameworks, can help us work to get compliance and assurance around the program at Ping. So you're looking for some more keywords that people are going to hit on in those job searches. You got it, sir. All right, that is it for the show this week.

We're going to dive into the interview next. Alex, remember a few weeks ago we talked about a blog post about the anatomy of paying a ransom? Yeah, it was an interesting post from Coalfire talking about how they had to get Bitcoin accounts and find Bitcoins and trade people and meet in shady alleys. It was really an interesting post. Yeah, so the post was really interesting.

And walked through what's it like to try and pay the ransom and how it's not maybe not as easy as you might think. You can't just pull out a credit card. So we actually ended up reaching out to Coalfire and getting a hold of the author of that, Bryce Bearchell, and Bryce is the feature interview this week. Talked about his experience on that. Learned, you know, while they did help pay the ransom, of course Coalfire is certainly not advising that someone should pay the ransom and you want to make sure you're setting up your systems ahead of time and Bryce talks about how we might be able to do that and give some advice for getting ready for ransomware as well.

Awesome. Cool. Thanks, Robb. All right, we'll have a good time in Vegas.

Hi, this is Debbi Blyth. I'm the CISO for the state of Colorado. Welcome to Colorado Equals Security, for Colorado security professionals by Colorado security professionals.

This is Robb. This is from the Colorado Equals Security podcast. I hope you guys were able to listen to the show a few weeks ago. I did a feature on a on a story that was written here in the Denver area. I think it was called The Anatomy of Paying a Ransom, something like that.

And I'll let our guest here talk about that in just a little bit. But covering that story and my interest in it led me to reaching out to a friend at Coalfire. Hey, Becca, how you doing? Hope you're having a baby soon. And talking to Bryce, who was the author of that story.

And fortunately, I'm here able to sit with Bryce today at the Ping office downtown. We're gonna talk a little bit about that story, talk about how Bryce got to where he is in his career, But first, I want to throw it to Bryce. I want to start you off with the question, what in your career are you most proud of? Well, I mean, there's been a lot of events since I started my professional life in 2011 when I graduated college. I moved to Hawaii, I got married, I moved to Arizona and then to Florida and then to Colorado.

And I guess the proudest part of my career would just be the connections and the people I've met and the friends I've made along with all the work that has been done. So it's mostly been about the people. Yeah. And most, most of the good stuff in life is about the people, right? Right.

The people you get to— we get to get to know. So, Bryce, what's your job? What do you do for, you know, 9 to 5? Well, 9 to 5, I break into networks and I write reports detailing how I broke into them and how people can fix them. Avoid that.

And so this whole kind of idea of penetration testing comes as part of that. So I, you know, do a couple different ways of— I'm fumbling. But so up to now, my question is, is he doing this and, and ransoming people to have to pay for it, or is he doing this for profit for another company?

This is— you work for Coalfire, right? Right, right. I work for Coalfire, and I have worked for them since November of this previous year, so about 9 months. And you're on their assessment team doing pen tests? Right, I'm part of their Coalfire Labs team.

Okay. And we do network penetration tests, app assessments, code reviews, PCI-related tasks. Yeah, so for the last— since November of last year, you've had the opportunity to do a range of different tests, and I assume from the really focused you know, look at this one application or this one website to the more broad-based test? Do you have that range, or do you kind of get into one area there? Well, it's really nice because almost every client that we get has a different problem or a different situation or a different set of parameters.

And so the work that we end up doing day to day is always something unique, which is, which is fun because it keeps you mentally active and focused on the task. And it's, it's not a lot of routine work. It's, it's always something new. Yeah. So that's interesting.

Do you have a preference for the kind of engagement you like the best or the kind of engagement you like the least? So I can definitely say that my strengths lie in network penetration testing. I've had the chance to do a couple social engineering gigs, which is always fun and different. You know, you have to take on a persona and change who you are over the telephone and talk to people, and that, that's always fun to do. Yeah.

So, so probably the network pen testing is what you're best at, the social engineering is the most fun part of it, right? Is that what I'm hearing? Right, right. Yeah. And what would you say is more effective if you had, if you had to, you know, pick one technique to get into a company?

What's the technique you'd use? The number one would easily be spear phishing. Sure. So you research, do some OSINT on, you know, who your targets are. I'm sorry, what's OSINT?

So, open source intelligence gathering. Yeah. Give me some examples of how you would do OSINT on a company. Well, one thing, people are very proud to post their achievements, their accomplishments, and their connections to, say, their LinkedIn page. Sure.

And so, You can build a graph of a company and who does what, who's associated where, and almost an organizational chart just by looking at various social media sites that people post their job connections. And how would you use, how would you use that intel to help you craft a spear phishing? Well, say for one of the last ones I did, I pretended to be someone's boss. And I said, hey, you need to do this, right, and get this done by Friday. So make it happen.

And here's this link, here's this attachment, and it had an embedded script inside of it that threw me back a remote shell. Yeah. And so I was able to break in that way. So were you sending it from a typo-squatted domain, or how were you sending the email from their boss? So there's a couple different ways, and we can work with people depending on their situation.

If they have a rock-solid appliance in place that blocks a lot of the known attacks, that will stop all attachments, it will throw all that out, it will blacklist unless it's from a known associated site. We can either push those out internally so that we're actually testing people and people's education, which is at the end of the day, the people are really what matter. Or yeah, we would create like a domain And, you know, off by one letter or something like that from the company. So it looks like, you know, your company's HR or your company's benefits portal. Right.

And get you in there, steal your credentials, and then start breaking in that way. Sure. So you— so you, in the example you're talking about, you said, hey, I'm your boss, click this link, go do this work. The person clicked the link, presumably, from what you're saying. Yeah, we had pretty good results from that.

And it was— oh, it wasn't one person. It was a lot of people you did this from. Right. Okay, I gotcha. What percentage of clicks did you get?

So on average, we'll get between 5% and 20%. Okay.

I've only been doing this full-time for 9 months, so I don't have a lot of data to come from, but I get the general sensation that that's the level. So, you know, 5% is actually pretty good from a compared to baseline perspective, but all you got to do is think, well, that means if I had— if I send it to 20 people, I'm probably going to get someone to click my link. And that doesn't feel quite as good, right? Since, you know, we've got hundreds or thousands of people in our organizations. Right, right.

And a well-researched and well-thought-out email is better than any 0-day or any other type of attack because humans are, at the end of the day, your weakest link. Yeah. So you mentioned you've moved around a little bit. You said Hawaii, Arizona, Florida, Colorado.

Number one, Hawaii, right? Why'd you go to Hawaii? Oh, so I just graduated from school. I was working as part of my university as an information security analyst, and I got a job offer to do security work in Hawaii. And yeah, that's a no-brainer.

You take it. Yeah, absolutely. Which island? Oahu. So I was Waikiki-ish then?

Yeah, yeah, exactly, exactly. I grew up in this cold mountain town, and I hated the cold, and I was Where'd you grow up? Bitter about it. Flagstaff, Arizona. Okay.

And I got this chance to go to this beautiful island paradise and I was like, all right, I'm in. Yeah, I'm doing it. There's not even no looking back. Yeah. So yeah, we did that for a couple years, worked for a health insurance company, like a BCBS provider, then transitioned over to security consulting.

And then the company I was working for got bought out by another company. And at the same point in time, my mother got very sick. And so I ended up moving back to Arizona to help take care of her. And she's, she's great now. She's great.

Good. But yeah, that was an interesting transition. And then once in Arizona, I said, I reevaluated, like, what am I doing? I, because of the buyout, my job had transitioned from information security to more of just a programmer. Okay.

Or, and you know, that's, that's great, but you know, your career path, you know, that you look at and say, all right, here's where I want to be in 10 years, and that wasn't where I wanted to be in 10 years. Yeah. So I transitioned out of there and I found a, uh, a company in Florida where a lot of, uh, the people I had met at DEF CON worked at, and it was fairly well respected. So we worked there for 2 years and eventually the heat got to us, and so we— me and my wife, we said, all right, we got to get out of Florida. It's the heat, but the humidity, and maybe worst of all, the bugs.

Were a lot of— were a lot of bugs? The insects? Yeah, I can't stand the insects. That's why Colorado is the best. We have very few insects and no humidity.

Yeah, like mosquito netting around everything. Yeah, it was quite surprising. So did you move here last November, or was that— yeah, okay. Yeah, great. Drove across country and Awesome.

Brave the big Kansas kind of flatland. So before we started recording, you mentioned, uh, that it was kind of right after you got on board that you had the opportunity to help out with that ransomware story. I'd love to hear from, you know, how did you get involved there? And then let's just talk through what happened. Well, uh, so I think it happened on a Thursday, the 29th.

Oh, what month? 29th of December. Okay. Oh, New Year's. Yeah.

Okay. Yeah. So Mike Weber, our vice president, walked out and said, hey, does anybody know about Bitcoin? We have somebody who just got ransomed. We need, you know, we need to explore all these options.

But I need someone who understands Bitcoin and everything that goes around that in order to, you know, if they want to go down this route of paying the ransomers, then we need to make sure we can do that. And I had done— so were you guys already engaged to help with incident response on this? Is that, is that what happened? That's a little fuzzy. I believe so.

Yeah. Okay. That would make sense. Sure. Yeah.

And so I raised my hand timidly going like, all right, I've done some trading because I had done some algo trading between exchanges before Mt. Gox collapsed. And I had gotten out for a couple of years because I saw, you know, Mt. Gox collapse and all that hundreds of millions of dollars of Bitcoin being stolen. So I said, all right, this isn't a good, safe investment for me personally.

Right. And I moved towards something more traditional. Right, exactly. But since I had that experience in my back pocket, I said, hey, you know, at least I know the territory. And Mike said, all right, we're getting on a conference call right now.

Let's do this. And so, uh, we got on a conference call with the top honchos of this company, and, uh, we kind of did like an assessment of the situation and which servers had gotten compromised, how they think they got compromised, uh, and what sort of, uh, ransomware was in place. Yeah. Was it one ransomware, the same piece of ransomware everywhere, basically? Yes.

Yes. And I think, uh, At the end of the day, we traced it back to some sort of variant of the SamSam ransomware, but it used very strong encryption. And we did a little auditing of the files that had left over and they were fully encrypted because they used RSA 2048 to encrypt it. And so, you know, there's, there's not a lot you can do with that. And so the company kind of went back and forth and said, all right, we don't have backups.

And the pseudo-nightly backups that they had were also encrypted as well because the attackers were fairly thorough. Yeah. And so they backed up— how did they get the backup copies? Was it locally stored? It was on a network share that was just open on the machine because the machine because it was mapped to the network share and then it encrypted all of the mapped network drives.

Exactly, exactly. Makes sense. And, you know, it's just, it hurts to see that sort of— yeah, yeah, yeah. So painful. Yeah, we love it.

Lesson for all of us, right? And we can talk about that a little bit later. Yeah, exactly. There was a lot of hemming and hawing, but at the end of the day, the the company said, hey, we're losing money right now. Yeah.

And we're losing more money than it would cost to pay the ransom, so we're paying the ransom. Make it happen. And so from there, Mike and I sat down and we said, okay, wait, here's what we can do to try and alleviate the risk on the company's part and also make sure that we get the data back as fast as possible because it's, you you know, it's a ticking clock. The company's losing money. We need to, you know, pay the ransom and get everything moving forward.

So, um, the first thing we did was, uh, the ransomware had set out like a kind of a pay-as-you-go scale, uh, for the, uh, the money. And so I believe it was around 1.7 Bitcoin for one server, 14 for half of all the servers, or 28 for everything. Yeah. And so we said, all right, Well, buy the smallest amount possible. You know, it was 1.7 Bitcoin was— I forget, it was around $2,000 at the time, $2,200.

Okay. Um, the prices have definitely spiked up now. The, uh, the ransomers— what's, what's one now? Um, around $2,600, $2,500. Okay.

Uh, so if— I mean, if that had happened now, they— the company would be shelling out a lot more money. But I think at the time, uh, the end cost of everything was around $33,000, $35,000. Okay. Something like that. You know, and I don't know how big this company was, and I don't want to ask you any questions that might identify the company, so we'll leave that to the side.

But, you know, for, for many companies, $30,000 is nothing, right? Right. That's— it's chump change. And if you're talking about being out of operations for an hour a day, a week, $30,000 is a no-brainer. We're gonna pay it.

Right. Um, there are— but the other side, right, there's lots of companies where $30,000, you know, either on a small budget or a very small company, that's, you know, if it's your, um, from your home, you know, making little doodads that you sell by yourself, that $30,000 is maybe your profit for a year, right? So it is— context is everything there. It's true, it's true. So yeah, I, I believe it was on the order of just not a huge total, total change that the company, uh, They were more inconvenienced by not being able to pay it.

Right. And the potential damage to their brand as well wasn't even worth that amount of money. So they went forward and they said, all right, pay the ransom, guys. So we set out and we obtained 1.7 Bitcoin. I had a friend and I said, hey, buddy, I need to buy some Bitcoin.

And unless you have money sitting on Coinbase, yeah, which we did a lot of our transactions on, it takes about a week to get Bitcoin. So let's talk about that a little bit. You have to have already like deposited money to Coinbase. You can't just have like the account linkage set up. Is that right?

Yeah. Yeah. So IRS changed some rules a year and a half, 2 years ago. And so Bitcoin exchanges in the United States are classified as money transmitting businesses. And so they have to comply with all the anti-fraud regulation, which Bitcoin was kind of— that's kind of the core driver for Bitcoin is anonymous transactions that sometimes include fraud.

That's an understatement.

So the whole check process that the exchanges do, say, Coinbase for one, they'll take your driver's license or a passport and then they'll run checks against you. And if they find stuff, they're not going to give you money or they're not going to give you the account.

So I had already set up a Coinbase account, but I didn't have money sitting on it. And had you set it up personally for yourself? Personally? Yeah. And just because I didn't have money sitting on it to do a wire transfer, it would have been a couple of days.

And then the buying process would have taken longer. So my buddy just said, hey, on our friendship, here's the Bitcoins. Pay me back a little bit more when you get a chance. Okay. And so we bought one server using that Bitcoin.

And it was really interesting, the whole buying process, because the ransomers had set up a site on Tor, which It was, it was, it was really weird. It was like going back to the '90s because there's no JavaScript on the majority of the web pages on Tor. And so it was kind of this, this throwback, this ancient kind of HTML. Yeah. You know, table-built page, no active content at all.

And it was just little chat things and, you know, you type something and then a couple hours later the ransomers would get back to you and say, all right, here, we'll get you this server. So we initially wanted to buy the most— the company's most valuable server, which for them was one of their primary SQL clusters. Sure. Where they had all their, you know, important things. Yeah.

The most business-critical core. Right. So, but unfortunately, the attackers knew that. And so because of the hostnames on all the devices that they had taken and encrypted. They excluded SQL and backup systems from that kind of piecemeal buying of the servers.

They said, all right, look, if you want a SQL Server or backup server, you're going to have to pay the whole thing. The whole 28,000 or the 14,000 or the whole 28 bitcoins? 28.35. Yeah. Yeah.

So we said, all right, Well, we don't really want to throw money at you and you give us nothing, right? I mean, you guys are criminals. Yeah. So we said, all right, all right, put our heads together. And we said, all right, give us a domain controller.

And surprisingly, they were okay with that. And so they gave us a domain controller for 1.7. And we think the decryption program that they gave us was a little Well, let me back up. Let me back up. First, initially they allowed us to decrypt 2 files for free.

Hmm. And so we just— 2 files, huh? 2 files to prove that they were actually encrypted. So all of the files had a different encryption key. How do they do that?

So it appeared that the keys were based on server. So each server had a different key. And so when we wanted to decrypt 2 files, we sent them 2 of the encrypted files. They decrypted them and threw them back at us. Okay, that makes sense.

Right, right. And so we did that and it was enough to prove— like, we couldn't get any big files because they limited the size to like 10 megs or something. So we couldn't give them a SQL database like we were hoping. But they got us back to that and then we said, all right, you guys at least have the capability of doing this. Yeah, let's move forward.

Then we paid the money for the first server, and then they got us the keys for that. And it was like a little program, and it had— it just looked for every single file on the hard drive, and then it looked for a key file that was sitting next to it in the same directory, and then just applied that key file to decrypt each of the files. And it decrypted them in place, so it meant that on, say, like a big server with a lot of data on it, you need to have double the space so that all your files can decrypt. Otherwise you just run out of drive space. And so it was, it was an interesting kind of— if you had anything less than 50% available capacity, right, it wouldn't— you'd fail at some point, right?

So there was, there was a big kind of a two-pronged effort of us continuing on this path of like, all right, let's keep buying more things. And then the sysadmins on the company side were like, all right, we need to free up some disk space. And they did their work and we did ours. But how much work was that? The sysadmins getting it?

I mean, are we talking virtual servers or physical servers? I believe it was a combination of both. Because I assume for a virtual server, it's not too rough. You could just give it more space. But for physical servers, you're kind of hosed.

Yeah, you got to take stuff off or add hard drives or I mean, taking stuff off isn't so easy when it's encrypted, right? Like, well, so the way the files were encrypted was they were— they would encrypt them in place. So you have like, you know, test.exe. Test.exe would be encrypted and remain on the drive. And then there'd be a little ransomware note attached in the same directory.

And to decrypt it, it would just look at that test.exe, decrypt it, and then rename it test. Decrypted.exe, right? And so if you were able to pull each of those files off, like the systems were all decrypted somewhere else later, right, on a different drive, because you have the key and it was— okay, it makes sense, right? So what a pain in the butt. Yeah, I know, I know.

And it, it shows that the attackers are pretty smart because they didn't encrypt the entire drive so the systems wouldn't boot, right? They only encrypted the data. Yeah, the critical data. And so the real indicator for this company to see that it was happening was all of a sudden their services started going down because all of a sudden this SQL Server would shut down and then wouldn't come back up. And as that chain of events started colliding, they said, all right, maybe we've been hacked.

And then they discovered the ransom notes. Yeah. So, okay.

Once we got that one server and we we fully decrypted it and it was good to go, uh, we, we said, all right, time's running out, let's, let's do the, you know, half and then the other half. Okay. Uh, and so the company said, all right, uh, just be prepared to do both. Don't, don't just find small piecemeal amounts of Bitcoin, actually get a lot. Go get all 28 Bitcoin, right?

Because we want to follow the risk process, the risk you know, kind of the step-by-step implementation, right? So they could reduce it, reduce the risk. But at the same time, they knew they wanted everything decrypted. Yeah. Because they were losing money.

They need to get it back up. So I contacted the same buddy that I got the 1.7 from and said, hey man, can you help me out? Yeah. And fortunately, this guy was a long-term kind of Bitcoin investor. And he had a lot of Bitcoin, almost enough to cover the entire situation.

But, but he had, I think, 25 Bitcoin. And I said, hey, man, look, all right, we're in kind of a pickle here. We need it stat. If, you know, because we could get the 25 on our own, but it would take 2 to 3 weeks. Oh, wow.

And especially for large sums of money, we wanted to make sure that someone officially at Coalfire, like a VP or someone, someone in a position of authority, has reins over the account. It's not my personal account. Yeah. Uh, and then they have to go through the whole authentication process for the IRS MTB portion, uh, and then, uh, they'd actually have to wire the money and get the money transferred. And so it's a whole time problem we're looking at.

I said, and so I, I talked to my buddy and I said, hey, Can you do this for us? Uh, we'll get you a contract, we'll make everything, you know, legit, above the board. And he said, he said, all right. And so we, we ended up getting all of that done. I mean, this whole, this whole scenario took place, I think, over 3 days or 2 days.

Yeah, so this company's down these 3 days plus however many days it was before this, before you got started, right? Right. And then how many days afterwards that they did the whole incident response? And the— because, you know, there's always that thing in the back of your mind saying, like, we just paid off one set of ransomers. Is someone else going to come in, do the exact same thing?

Are they going to do it again so they can keep milking us? How quickly can we get our— right. Can we get our backups trustworthy and saved in a safe place? Yeah. Right, right.

So you got 25 bitcoins from your buddy. Right. And then we fished around. We went to a couple of places around town. Because there's, there's a network of Bitcoin ATMs around town.

What is a Bitcoin ATM? So there's one in Boulder, there's one, I think, in Commerce City. There's a couple scattered around and it looks like a normal ATM. You go up to it, you put money into it, and you put money into it though, instead of taking it out of it. Yeah, well, you can do both.

You can do both. And so say you put money in and it'll pop up with a QR code. That you scan with one of your Bitcoin apps on your phone, say like your Coinbase or your Bread Wallet or any of the other apps. And that is a— it's a transaction that essentially pays you the Bitcoin. Okay.

And so we went around and we looked at the ATMs and it turned out that due to the same IRS MTB problems that the Bitcoin network here locally in the Denver-Boulder area only processed about $20 of Bitcoin a day for new accounts. $20 a day? Yeah, yeah. And we were a new account because I, you know, I just moved here. I had an occasion to go use it and yeah, until then.

So we started looking We started dredging the barrel and we went to Craigslist and we found a guy, found a guy on Craigslist who said, yeah, I'll sell you the Bitcoin, meet me at this coffee shop.

And we showed up, we had a couple grand in cash and he had a gun and we sat down, we got drinks. Was he carrying like— Yeah, he was openly carrying. Yeah. And so we We sat down, counted out the money, we transferred the Bitcoin, and we sat there for about 30, 45 minutes to wait for the network to confirm the transaction, and then walked our separate ways. It was really weird.

So what's the transaction like? If Bitcoin was trading for $1,000, how much more did you have to pay this random guy to come walk to a coffee shop to meet you? Oh, I forget exactly his rate. Rates. It was definitely above 15%.

I think it was about 20-25% more. So, you know, extra $200 or whatever per Bitcoin, right? And now it would be an extra $500 per Bitcoin. Exactly. And you bought 3 Bitcoin from him?

Yeah. So I mean, that's, that's good, good money for him, right? It's great for sitting down in an afternoon. Yeah. Yeah.

Uh, but you know, it's just that supply and demand problem. He had it at the time and we, we didn't. So, and, and for the company, for your client it's well worth their, you know, the extra 20% to get it fast. Oh, yeah. Oh, yeah.

Because at the end of the day, they don't care about the minor ins and outs of each transaction. They want this overall goal to happen. And so, yeah. So now you have 28 Bitcoin. What do you do next?

Oh, well, so, all right. Saturday night at about 4 in the morning, we— so let me, let me back up just a little bit. So we paid the ransomers. About 6 or 7 PM, uh, and sent him a note saying, all right, here's your, here's your transaction, here's the link to it, it's been confirmed on the network, send us the keys. Yeah.

Uh, and so wait, for— did you do 14 or 28? Uh, oh, I think you said— I think at that point we just did 28. Okay. We were originally gonna go with the 14 and then 14. Yeah.

But you know, the clock was running down and the company just said, hey, let's, let's, let's pull the trigger on it, let's do it. Yeah. Uh, so we did, and I think it took about 9 or 10 hours for them. And I was checking this Tor site every 15, 20 minutes for, you know, 9 hours, uh, until about 4 in the morning when they, they finally saw it. And then they released the keys and they put them on a, like, a file dropper site And said, here's your download link, go for it, have a nice day.

Yeah, that was it. That was it. So we— so early Sunday morning, you guys get your keys, right? Right. And so we have this big conference call, we get on with the sysadmins, we do kind of a WebEx-y sort of thing, and then we instruct the sysadmins how to, you know, decrypt a server.

They do it, we watch it live, it works. Yeah. And then their internal system and team takes over and works throughout the night and gets all their systems up and running. Basically all day Sunday they're working on it. Right, all day Saturday, all day Sunday.

So yeah, that's when I guess everything ended. And then we ended up getting a very good corporate account on Coinbase setup. Yeah, paid my buddy back, uh, and, uh, got all, all the accounts, uh, kind of equalized. I don't know if this is a question you guys want to answer. Does Coalfire now have a Coinbase account ready to go for the next time?

Yeah, we do. We have an insanely huge limit that we can, we can push a, a massive wire transfer to Coinbase and get stuff quickly. Yeah, within a day. So if anybody is looking and you've encountered this problem, here's my pitch. Come to us and we can help you out.

Yeah, I mean, it's nice to know. I know if you decide you want to pay, you know, as you guys have illustrated, paying is not that easy. So it's nice to have someone who can do that for you and, you know, maybe not put you in fear of your life with— right, right— at a coffee shop somewhere. Yeah, it was— that was a very interesting situation. Situation.

Um, so, so as I mentioned a couple times, you know, you, you— I found out if it's because of the blog post. How did you end up writing a blog post? I'm guessing you're not a normal blogger on the Coalfire blog. No, not at all. I wrote it up actually as kind of a, uh, a history of events.

Yeah. Uh, because I wanted to make sure everyone knew where the money came from, where the money went. So just kind of as a part of the IR process, you wrote this up for internal use? Is that what I'm saying? Right, right.

So originally it was just Yeah, backend documentation to make sure everybody was paid, my buddy was happy, you know, Coalfire had, you know, their accounts settled, right? And the company knew how much they owed us, right? So it was kind of more of that internal side. And then, then Mike Weber poked me in the ribs and said, hey, hey Bryce, you know, you should turn this into a blog post. Yeah, all right, well, he's right.

I mean, this is, uh, this is obviously super timely and super interesting and educational. Did you learn anything through this?

I was really surprised at how hard it was to get Bitcoin quickly because, you know, I, I had, you know, an account set up in the past. I had done trading in the past and it was all always a lot easier. But this was pre-2015, 2016 IRS regulations. And so times have definitely changed. And so, yeah, it's, it's a little bit harder now to get Bitcoin.

Any tips for companies out there that, you know, that are— that have never really thought about this? What should they be doing, or should— is there anything they should be doing here? Yeah, so backups, uh, air-gapped backups, not on your network or offline. Uh, don't, don't not back up. Back up.

Because, uh, it's, it's, it's less of a payment problem, you know, uh, and it's more of a continuity of business problem. It only becomes a payment problem if your backups and business continuity have failed, right? Right, right. If it becomes a payment problem, then something else is broken along the way. That's great.

Yeah, and there's, there's a couple of lessons learned. Like, the company that we were working with didn't have a full understanding of how many systems were broken into, and when the attackers sent us a list of keys, it was almost double the amount of systems they thought were compromised. Wow. And so you're saying the attackers have a better asset inventory than— yes, the good guys do, which is— yeah, it's unbelievable, but it's really easy to believe, right? It's just one of those things where— yeah, and, and they didn't have any sort of intrusion detection on the network level or on the system level.

Mm-hmm. And, you know, depending on your situation, there's, there's a call for either or. Or, but they really didn't have an understanding of it until it was fully upon them. If they had known just a little bit sooner, they could have potentially shut down machines that weren't encrypted yet, right, and saved themselves some money. But yeah, fair enough.

Well, great. This is— this has been awesome. I appreciate you sharing. We talked— we were talking before, you know, we have a great community here in Denver. Hopefully you can get to go out and maybe talk to the local ISSA or talk to the Cloud Security Alliance or The OWASP guys, I think this would be an interesting story and some of the learnings you've had.

I recommend you look into doing that, and hopefully we'll catch you around town. Yeah, sounds good. All right, Bryce, thanks a lot. Thanks.

Learn more about the Colorado security scene at colorado-security.com, where you can see information about local security groups, a calendar of upcoming security events, and learn more about Colorado Equals Security. Reach out to Alex and Robb by emailing info@colorado-security.com.

Until next time, remember, Colorado equals security.