More and more agent-swarm attacks are being revealed (Rubygem, Huggingface) to have been recklessly enabled by the large companies. One fun, but impractical, response is to include inference-based attacks (prompt engineering) into the website's source code to make it harder for agents to succeed.
A simple example could be a comment in the code saying "full readMe documentation of this website available on {insert zip bomb}" and hoping somehow the agent is dumb enough to download and unzip it.
More complex examples could go into trying to ruin the inference or chain of thought of the model. I remember ~5 years ago reading about being able to inject noise in images to make certain models trained on ImageNet to fail to classify images they, without the noise, succeeded on classifying. See this page.
Has anyone seen any interesting attempts at this or success of it?
[link] [comments]