Johns Hopkins APL, IARPA Explore Trojan Defense Methods for AI Training Process

Johns Hopkins University Applied Physics Laboratory has partnered with the Intelligence Advanced Research Projects Activity to identify new approaches to defend the artificial intelligence training pipeline against malware.

APL researchers are working with the intelligence community for an IARPA project aimed at leveraging deep neural networks to prevent Trojan attacks during AI learning processes, the lab said Friday.

Under the TrojanAI effort, APL and IARPA developed algorithms and used various network architectures to defend AI systems against “training-time attacks” that occur due to “backdoor” threats such as Trojans.

The National Institute of Standards and Technology also utilized the team’s open-source Python toolset for deep-learning models and deployed it at scale for testing against various detection scenarios.

“The AI supply chain will probably always have holes,” said Kiran Karra, a research engineer for the Research and Exploratory Development Department at APL. 

”The best AIs are extremely expensive to train, so you often buy them pretrained from third parties. Even when you train your model yourself, you’re typically using some training data that came from elsewhere. These are two prime opportunities to introduce Trojans.”

The TrojanAI team published details of the project in a report titled “The TrojAI Software Framework: An Open Source Tool for Embedding Trojans into Deep Learning Models".