Can Claude Do My Tax Return? Capabilities and Limits of AI

In a previous discussion, I explored Claude’s Computer Use capabilities with browser automation. In this blog post, I'm giving Claude Computer another try. This time, I want to see if it can fill out a tax return using data I have stored on my hard disk. I'm intentionally not giving it the data in the prompt; instead, I want to see if it can:

The Challenges:

Read the form and determine the required fields.
Find the relevant information to fill in those fields from the data on disk.
Fill out the form accurately.

For this experiment, I've selected a tax return form, specifically the IRS Form 1040:

https://www.irs.gov/pub/irs-pdf/f1040.pdf

And to make things easier for Claude (and because I initially had trouble with PDF editing), I've generated a simplified web-based version of the same form:

Why the web form and not the PDF editor?
I tried using a PDF editor first, but Claude CS didn't manage to fill it out. I suspect this is more of a driver or compatibility issue. So, I switched to a browser-based version for this test.

I previously did the same experiment with Formix PDF (see screenshot below), which uses Claude afaik, and the results were not bad. I'm wondering if using Claude Computer could do even better.

For this experiment, I created some sample tax data and saved it in the /.antropic folder, which is shared between Claude Computer and my Mac. Here's the script I used to generate the data:

1. W-2 (Employment Income)
- Employer: Tech Corp Inc
   - EIN: 94-1234567
   - Address: 742 Corporate Way, San Francisco, CA 94101
- Employee: James Wilson
   - SSN: 453-89-1234
   - Address: 742 Evergreen Terrace, Springfield, CA 94101
- Income Details:
   - Box 1: Wages, tips, other compensation: $95,000
   - Box 2: Federal tax withheld: $18,500
   - Box 3: Social Security wages: $95,000
   - Box 4: Social Security tax withheld: $5,890
   - Box 5: Medicare wages and tips: $95,000
   - Box 6: Medicare tax withheld: $1,377.50

2. 1099-INT (Interest Income)
- From: Mega Bank
   - EIN: 95-9876543
   - Address: 100 Finance Blvd, New York, NY 10001
- Recipient: James Wilson
   - TIN: 453-89-1234
- Income Details:
   - Box 1: Interest income: $1,200
   - Box 4: Federal tax withheld: $336

3. 1099-DIV (Dividend Income)
- From: Investment Firm LLC
   - EIN: 93-4567890
   - Address: 200 Wall Street, New York, NY 10002
- Recipient: James Wilson
   - TIN: 453-89-1234
- Income Details:
   - Box 1a: Total ordinary dividends: $3,500
   - Box 1b: Qualified dividends: $3,000
   - Box 4: Federal tax withheld: $980

4. 1098 (Mortgage Interest)
- From: Homeloans Bank
   - EIN: 91-2345678
   - Address: 300 Mortgage Row, Chicago, IL 60601
- Recipient: James Wilson
   - TIN: 453-89-1234
- Details:
   - Box 1: Mortgage interest received: $12,000
   - Box 10: Property taxes: $4,200

5. Property Tax Statement
- From: Springfield County Tax Collector
- Property: 742 Evergreen Terrace, Springfield, CA 94101
- Owner: James Wilson
- Details:
   - Parcel: 12-345-678
   - Annual Tax Amount: $4,200
   - Payment Schedule:
      - First Payment: 11/01/2023 - $2,100
      - Second Payment: 02/01/2024 - $2,100

6. Charitable Contributions Receipt
- Organization: Springfield Foundation
   - EIN: 92-3456789
   - Address: 800 Charity Lane, Springfield, CA 94101
- Donor: James Wilson
- Contributions:
   - March 15, 2023: $1,000
   - July 20, 2023: $800
   - December 15, 2023: $700
   - Total contributions: $2,500

Having all the data ready, let's go!

The Initial Prompt:

Here's the initial prompt I gave to Claude Computer: "fill in the form in firefox using the fields in /.home/computeruse/.anthropic"

Initially, it started a bit off-track, asking for information that I had already provided in the prompt. So, I simply told Claude to re-read the initial prompt.

Claude Gets to Work:

And it did! It figured out what it needed to do. First, it checked all the files in the specified folder: