05 June 2023
By Maggie Engler, Technology Advisor at First In
No area of technology has attracted more attention over the past six months than generative AI. That fertile period, sparked by OpenAI’s release of ChatGPT in late November 2022, included major product launches based on large language models (LLMs) from Google, Microsoft, Facebook, and a host of startups. As a result, founders, corporate leaders and investors alike have sought to participate in the explosive growth of generative AI. The use cases of accelerating generative AI capabilities are growing rapidly and the implications for cybersecurity, as well as security technology more broadly, are emerging.
The next generation of LLMs
There is no denying that today’s LLMs are impressive, with capabilities including text and code generation, question answering, and summarization. For the most part, however, these models are effectively dialogue agents, requiring the user to provide prompts and do whatever they need with the output. Experiments in building more autonomy into these tools are fairly early — Auto-GPT, which uses GPT-4 and web retrieval in a loop to attempt to accomplish goals on its own, has documented problems with recalling its previous results and chaining together tasks. LangChain, a popular framework for developing applications with LLMs, takes as its guiding principles that the “most powerful and differentiated” applications will be both data-aware (able to connect to external data sources) and agentic (able to interact with their environment). Agents with the natural language abilities of LLMs and access to data sources like the web, trained via reinforcement learning could ultimately take on bigger and more complex objectives, such as writing and deploying software or performing scientific research and reporting results. In addition, we can expect to continue seeing iterative improvements that allow for these models to have longer memories, accept longer inputs and outputs, and run more efficiently without a decline in performance.
The near-term use of generative models in security
Generative AI makes a variety of tasks in the security space easier, but, at least in the short term, accelerated AI capabilities will be more useful for defense than offense due to the increased marginal benefit of using AI for defensive purposes relative to offensive ones.
While there is understandably a lot of fear and excitement around generative AI in cybersecurity, there are reasons to doubt that it will be transformative to the industry in terms of offensive capabilities or threat vectors. Exploring how adversarial attackers might leverage LLMs provides an illustration. A commonly held concern is that adversaries could use LLMs to write exploits for common vulnerabilities, a worry that has already surfaced in several news articles. They might also use LLMs to write content for social engineering. Similarly, people who wanted to spread a particular narrative online could use LLMs to give voice to bots and manufacture the appearance of consensus around a given topic. Fears are particularly centered on whether LLMs could help scammers and social engineers scale damaging efforts significantly. However, none of these possibilities are new capabilities; arguably, it’s still easier to find code snippets for writing exploits online than it is to get ChatGPT to write one. It is also already surprisingly cheap to operate bots and content farms, and existing bot detection relies more heavily on behavioral signals than text indicators. AI will certainly render some types of defenses obsolete — there are already examples of biometric voice identification systems being broken with synthetic audio — but it remains unclear whether there will be the momentous sea change that one might expect and some are predicting.
On the other hand, consider how cybersecurity and other defensive systems will be impacted. As mentioned above, LLMs such as GitHub’s Copilot have already shown tremendous promise as coding assistants. LLMs’ current limitations, including the tendency to hallucinate inaccurate information, mean that generative AI is still insufficient as a replacement for engineers, but a reasonable first pass can be generated for many functions, documentation, and test cases. One particularly exciting application is the use of generative AI to create synthetic data, which could be used to evaluate and improve existing supervised learning systems. As one example, existing natural language processing systems, such as spam or phishing email detection models, can use LLMs’ incredibly high degree of natural language understanding to increase accuracy while requiring less training data. In practical terms, traditional phishing classification models might require hundreds of messages employing a new type of scam to begin to detect similar messages, whereas an LLM-based classifier might be able to learn to detect the scam with a natural language description alone (“flag any email not from hr@company.com that asks the recipient to send them confidential information”) or a description with just a few examples provided. The ability to specify and change policies enforced by AI in natural language will change the machine learning (ML) development lifecycle significantly, with significant impacts on defensive cybersecurity.
Security and tooling for modern ML products
Substantial barriers to LLM productization remain, particularly within an enterprise setting, with data privacy and security being two of the primary concerns. Companies must be able to ensure that, if they are calling external APIs for LLMs, their data remains private and protected. MLOps and MLSecOps are both relatively new areas of specialization focused on securing the AI and ML lifecycles, which will be an ongoing challenge, as the requisite computing clusters for training are an attractive target for cyber adversaries. Other types of tooling that will become increasingly important for LLM adoption are those that improve the ease of customization, like vector databases, which have been used to add long-term memory to existing LLMs.
There is still much to be explored and discovered in generative AI. At First In, we are paying close attention to the developments of generative tools and the broader ML ecosystem and infrastructure. The full scope of what these models will become capable of in the future remains unknown, but it seems clear that they will cause major changes across sectors, and understanding these models — including their risks and shortcomings – will be key to capitalizing on this shift in security technology.