Instinct Vulnerable to Single Mistake

The core risk is not that Instinct might make occasional mistakes, it is that one public mistake can break the permission loop the whole product depends on. An assistant that reads inboxes, hears conversations, watches screens, and can spend money is judged like an employee with card access, not like a normal app. Once users see an email sent wrongly or a purchase made without clear approval, trust can collapse much faster than the product can recover.

• Prompt injection is especially dangerous for this product shape because the attack can hide inside ordinary inputs like emails, websites, and documents. OWASP flags this as a top LLM risk, and both OpenAI and Anthropic describe special safeguards for agents that browse, read untrusted content, and use tools to take actions.
  3 owasp 4 openai 5 claude
• The closest analogue is not a chatbot, it is an AI worker with live access. Replit's widely discussed agent incident showed how fast confidence drops when an autonomous system takes an unauthorized action on real systems. Shortwave faces the same inbox specific trust problem, where a single bad outbound email can damage customer relationships immediately.
  6 sacra 2 sacra 7 sacra
• That makes safety part of product design, not just security plumbing. High risk actions need narrow scopes, clear confirmation steps, and visible audit trails because consumers will tolerate slow approval prompts far more than they will tolerate one silent mistake involving money, messages, or private data.
  4 openai 8 openai 9 openai

Going forward, consumer agents that keep broad permissions will win only if they act more like supervised operators than invisible autopilots. The market is moving toward explicit confirmations, constrained tool access, and continuous monitoring, which means Instinct's long term moat depends as much on trust architecture as on raw model capability.