Spree of multimillion dollar hacks creates booming business for blockchain security experts

Even as cryptocurrency markets face economic turbulence, there’s one segment of blockchain-based industries where business is booming: blockchain security.

A boutique industry of auditing firms formed over the past few years to deal with the emerging technology now boasts up to a year-long wait time to even begin working with customers and a growing list of job openings they can’t fill quickly enough.

And investors are flocking to get a piece of the action, too, pumping millions of dollars into firms that promise to help safeguard an increasingly fragile cryptocurrency ecosystem.

From the outside, the race for security seems like a long overdue course correction for an industry now plagued by near-weekly multi-million dollar hacks. However, security experts in the industry don’t all necessarily see the boom in business as an unmitigated win for the industry, they tell CyberScoop. Instead, they say it points to a much deeper challenge for the industry: cultivating the kind of security talent needed to keep a growing financial industry under the constant threat of hacks safe.

“It is not a good thing that there is a dependence upon external consultants for core competency required to build blockchain software,” said Dan Guido, founder of security firm Trail of Bits.

Crypto companies hire Trail of Bits to independently audit their code for vulnerabilities, a process that Guido emphasizes provides some reassurance to the company but does not constitute the same level of safety of full or ongoing security reviews.

While experts like Guido adamantly advise that companies have other security processes baked into their development and review processes, external audits have become a crutch for an industry hobbled by a lack of blockchain security experts.

“You have a talent shortage in cybersecurity, in general,” said David Schwed, chief operating officer of blockchain security firm Halborn. “And then a subsection of that is this new and emerging technology where it requires a different type of thinking than traditional cybersecurity professionals.”

Blockchain projects offer distinct challenges for security professionals. Foremost, many are written in newer and less common coding languages such as Solidity, narrowing the pool of individuals who can audit the code. Unlike many other systems, which are designed to be closed off in an effort to thwart attacks, the blockchain is public, meaning that hackers have an open book for vulnerabilities.

The bigger barrier to finding the right talent isn’t so much teaching people about blockchain as it is finding someone with the right mindset, Schwed says.

“I don’t want to say it’s a different level of paranoia, but that’s really is what’s required in this field,” said Schwed. “A transaction is immutable. It’s gone. That’s the important piece that they’ve got to understand.” Given the nature of some attacks, security experts must also understand how the technology works from the business side, he says.

Larger cryptocurrency companies take a similar approach in finding talent. Nick Percoco, the chief security officer at digital asset exchange Kraken, says that he looks for candidates who have both a strong security background and a hands-on interest in blockchain.

Percoco notes that while Kraken does use external audits for legal reasons, having an internal security team allows him to continuously test Kraken’s products for potential vulnerabilities. It also helps develop a company-wide security culture, something especially important as criminal and nation-state hackers increasingly go after employees of digital currency firms.

“It’s more than the systems, it’s more than the policies, it’s more than the software — it’s essentially a mindset that everybody in the company is put into,” said Percoco.

Both Schwed and Percoco pointed to bug bounty programs, in which independent security researchers report vulnerabilities for a reward, as another key avenue for finding new talent. Major firms like NFT platform OpenSea and Solana host their own hack-a-thons as a supplement to traditional audits.

As the industry waits on universities and traditional training programs to catch up to the needs of the blockchain industry, some security experts have taken a hands-on approach to nurture new talent.

“There’s the tragedy of the commons that happens with education and talent,” says Rajeev Gopalakrishna, a researcher who founded Secureum, an online learning community and boot camp for security experts interested in blockchain security. “Everybody wants to hire talent. But who is going to train them or build the content?”

Since 2021, hundreds of individuals have used Secureum’s online training program. Gopalakrishna says he knows of about 20 students who have gone on to full-time work with auditing companies though many have taken the skills to do more hobbyist work like bug bounty programs. Trail of Bits also offers an apprenticeship program for security experts interested in blockchain.

Human intervention isn’t the only answer. Experts also pointed to advancements in automated tools that can help developers with more basic security functions. But such tools will never be a complete replacement for human expertise, says Guido. His firm found in a study that automated tools caught only roughly 50 percent of vulnerabilities in blockchain projects.

Of course, solving the blockchain security skills gap will only help security in the industry insofar as the growing number of crypto startups take advantage of it. The rapid development cycle of blockchain projects and the boom and bust nature of the industry means there will still always be developers who fail to prioritize security from the on-set.

“The overall security posture of the space was increasing, and then the bull market happens, and it’s really falling back to the way it was four years ago,” said Mehdi Zerouali, co-founder of security firm Sigma Prime. “And I think it’s just a matter of having so many more people joining this space, needing to potentially go through the same mistakes and realize the importance of security.”

Those mistakes are mounting. By one estimate, blockchain projects have lost more than $600 million worth of cryptocurrency from hacks in the second quarter of 2022 alone. And some of the biggest losses in 2022, including the record $600 million hack of Axie Infinity, were the result of traditional cyberattacks, not the exploitation of web3 technology. More recently, persistent attacks by North Korean hackers against cryptocurrency firms have rattled the industry and raised the concerns of the U.S. national security community.

“This has raised the stakes. It’s made the consequences of even minor failures much more severe,” said Guido. “And I just don’t think that many companies are prepared to operate in that kind of environment where they have a dedicated focus group of attackers that will stop at nothing until they achieve success.”

Those risks will continue to grow as blockchain technology develops and grows more complex.

“The average DeFi [decentralized finances] project we would look at one, two years ago has nothing to do with the average DeFi project that we would have now,” said Zerouali. “With innovation comes the question ‘How do you do so safely?’ It can be extremely difficult. So the more we progress the more complexity we’ll be facing, and the more risk we have to deal with.”

Correction 7/26/22: This story was updated because the original version incorrectly quoted Dan Guido, founder of security firm Trail of Bits, when referring to the use of external security consultants.