As AI agents become more sophisticated, the ways they interact with users are evolving rapidly. Understanding these interaction modalities is crucial for anyone building or working with agentic systems.

The Four Modalities

AI agents can interface with users through four primary modalities:

1. Text-Based Interfaces

The most common and mature modality. Tools like ChatGPT, Claude, and countless chatbots communicate through text. This modality excels at:

• Complex reasoning and explanation
• Documentation and code generation
• Asynchronous communication

2. Graphical Interfaces

AI integrated directly into visual tools and IDEs. Examples include:

• Cursor: An AI-powered code editor that understands your codebase
• GitHub Copilot: Inline code suggestions as you type
• Figma AI: Design assistance within the creative tool

3. Speech Interfaces

Voice-first AI experiences are becoming more natural:

• Wispr Flow: Voice-to-text with AI understanding
• Voice assistants: Siri, Alexa, Google Assistant
• Meeting copilots: Real-time transcription and summarization

4. Video/Visual Interfaces

The emerging frontier:

• Screen understanding and interaction
• Visual reasoning about images and documents
• Real-time video analysis

Why This Matters

Each modality has strengths for different contexts. The best agentic systems will likely combine multiple modalities, choosing the right one for each task.

For builders: understanding these modalities helps you design better AI-powered experiences.

For users: knowing what’s possible helps you leverage AI tools more effectively.

---

This is part of my AI Agents Explained series, where I break down the fundamentals of building with AI agents.