DEV Community

viky
viky

Posted on

26 Key Takeaways from Building hundreds of AI Agents

#ai

Introduction

We built over hundreds of AI agents under the new "agents as a service" model in the last several months. This document shares 26 key takeaways that we had to learn the hard way - lessons that cost us dissatisfied clients, time, and money, so you don't have to repeat our mistakes.

Agent Fundamentals

1. AI Agents Are Not Your Employees

  • Agents are neither automations nor employees
  • Automations: Every step is hardcoded with exact sequence known in advance
  • Employees: Have more autonomy than agents
  • Instead of thinking about agents in terms of roles, think in terms of SOPs (Standard Operating Procedures)
  • Typically one agent can handle one SOP well, while one employee handles 5+ SOPs

2. Start From Well-Documented Processes

  • SOPs (Standard Operating Procedures) make training agents significantly simpler
  • Well-documented processes contain most of what you need to train agents
  • Find onboarding materials and SOPs first before collecting data manually

3. Business Owners Will Never Build Their Own Agents

  • Even with agents that build other agents, business owners will still need specialists
  • Similar to how no-code tools created no-code developers, not the end of developers
  • AI agent platforms will spike demand for AI agent developers
  • Determining which agents to build is harder than building them

4. Business Owners Have No Idea Which Agents They Need

  • ~50% of initial client ideas are not the most valuable agents for their business
  • Consulting is a huge part of the service
  • Start by mapping customer journeys to identify automation opportunities
  • Don't assume client ideas are the best - use them as feedback only

5. You Don't Need 20+ Agents

  • Building too many agents makes systems more complex
  • More agents means:
    • Harder maintenance
    • More complex debugging
    • Increased costs
    • Longer response times
  • Start with the smallest possible agent, then add more as needed

6. Data With Actions Deliver Results

  • GIGO (Garbage In, Garbage Out) applies to AI agents
  • The biggest impact comes from combining data with relevant actions
  • Combining knowledge (e.g., Facebook marketing strategies) with actions (Facebook API control) achieves significantly better results
  • Scrape both internal and external sources to enhance agent performance

Agent Development

7. Prompt Engineering Is an Art

  • Prompt engineering is already a real job
  • As models become larger and smarter, prompt engineering becomes more important
  • Tips for effective prompts:
    • Provide examples - one example is worth a thousand words
    • Order matters - arrange prompts with most important parts at the end
    • Iterate and test constantly

8. Integrations Are Just as Important as Functionality

  • Don't over-focus on agent capabilities at the expense of integration
  • If it's not convenient for users, even powerful agents won't provide value
  • Integrate agents into the same systems employees use daily

9. Agent Reliability Has Been Solved

  • If an agent isn't reliable, it's a developer problem, not an agent problem
  • Pydantic (data validation library) can validate all agent inputs/outputs
  • With proper validation logic, agents can't take harmful actions
  • Pydantic allows building agents for any use case

10. Tools Are the Most Important Component

  • The three most important components: instructions, knowledge, and actions
  • 70% of work goes into building actions (tools)
  • Standard chatbots generate value through responses
  • Agents generate value through actions - they should do things, not just advise

11. No More Than 4-6 Tools Per Agent

  • More than 4-6 tools causes hallucination
  • Agents start to confuse which tools to use or the proper sequence
  • If an agent starts hallucinating, split it into multiple agents

12. Model Costs Don't Matter

  • If your use case makes sense, you'll almost always make tremendous ROI
  • Example: Process reduced from $300 and 3 days to $1-2 and 20 minutes

13. Clients Don't Care About Which Model You Use

  • Open source models aren't necessary
  • Businesses care about value, not the model providing it
  • For strict data privacy, use Azure OpenAI (runs models in private Azure instances)
  • OpenAI remains the provider of choice due to developer experience

14. Don't Automate Until Value Has Been Established

  • Don't automate businesses that don't exist yet
  • First establish the process manually to ensure it works
  • Development costs are the main concern, not model costs
  • Hire someone on platforms like Upwork to test processes before automating

15. Don't Think About Use Cases, Think About ROI

  • ROI formula: (Rate × Hours - Operational Costs) ÷ Development Costs
  • Rate × Hours = Employee hourly rate × Total process hours
  • Operational costs = Model costs + Server costs (typically negligible)
  • Example: $50/hour × 10 hours/week with $5,000 development cost = 5.6× ROI after one year

16. Agent Development Is an Iterative Process

  • Like data science competitions, testing the most variations often wins
  • Try different architectures and compare side by side
  • Build multiple variations when agents underperform to find what works best

17. Use Divide and Conquer Approach

  • Break complex problems into manageable tasks
  • Deliver solutions incrementally:
    • Find an agent that can work independently
    • Build and deliver that agent first
    • Only proceed after client confirmation
  • Automate by departments first before connecting systems

Scaling and Deployment

18. Evals Are a Big Deal (But Only for Big Companies)

  • Evals track agent KPIs and performance over time
  • Benefits for large companies:
    • Helps eliminate competition
    • Continuously improves solutions
    • New clients benefit from previous solutions
    • Enables future self-improvement
  • SMBs may not need evals due to lower request volume
  • Evals provide the last 20% of performance optimization

19. There Are Two Types of Agents

  • Pure agents: Fully agentic systems
  • Workflows: Processes with predetermined steps but agentic execution
    • Steps and sequence are fixed
    • Individual steps have agentic capabilities
    • Example: Lead research with specific search patterns

20. Agents Need to Be Adaptable on Feedback

  • Agents must interact with their environment and receive feedback
  • Add tools that allow agents to analyze their results
  • Example: Don't just build database update capability; add ability to read records to verify task completion

21. Don't Build Around Limitations

  • Models continuously improve - don't optimize for current limitations
  • Example: Complex systems built to avoid context token limits became obsolete when 128k context models were released
  • Avoid building obvious general use cases that major AI companies might develop themselves

22. Deploying Agents Is Harder Than Building Them

  • Building an agent might take 2-3 days
  • Deploying and integrating it takes another 3 days
  • Consider specialized platforms for agent deployment

23. Waterfall Projects Don't Work

  • Agentic projects are too agile for fixed scopes
  • Start with one-time fees but transition to subscription models
  • Work as a partner, not just an outsourced team
  • The goal is to automate business processes, not just build agents

24. Include a Human in the Loop for Mission-Critical Agents

  • Some agents have no margin for error
  • Add human review steps for high-stakes actions
  • Example: Having clients review marketing campaigns in Notion
  • Remove human review only after consistent approval and fine-tuning

25. 2025 Is the Year of Vertical AI Agents

  • Vertical agents specialize in specific use cases for specific industries
  • Benefits:
    • Easier to scale
    • Higher pricing potential
    • Better problem-solving for specific businesses
  • Start with horizontal agents, then identify patterns to create vertical agents

26. Agents Don't Replace People, They Help Businesses Scale

  • Business owners typically don't fire people after automation
  • Automation helps business owners think bigger and scale faster
  • Employees can focus on higher-level tasks they enjoy
  • Ultimately leads to a new age of abundance and prosperity

Top comments (0)