ChatGPT-4 outperformed 90% of developers

The title may sound bold and clickbait-ish, but it is what it is...

A few days ago I stumbled upon a post on LinkedIn (unfortunately, I didn't save the link) in which a programmer demonstrated his solution to a LeetCode challenge. The code was fine and the post got dozens of likes, which it undoubtedly deserved. I decided to feed the requirements to ChatGPT-4 and see if it could understand and solve the problem. The outcome was... interesting and I spent some time experimenting with the AI, modifying the prompt to see how it affected the results.

The Challenge

First, I would like you to take a look at the LeetCode problem and attempt to solve it. Take your time, give it some thought. Then, google it. Yes, search the internet for the best solution.

Given two integer arrays pushed and popped each with distinct values, return true if this could have been the result of a sequence of push and pop operations on an initially empty stack, or false otherwise.

Example 1:

Input: pushed = [1,2,3,4,5], popped = [4,5,3,2,1]
Output: true
Explanation: We might do the following sequence:
push(1), push(2), push(3), push(4),
pop() -> 4,
push(5),
pop() -> 5, pop() -> 3, pop() -> 2, pop() -> 1

Example 2:

Input: pushed = [1,2,3,4,5], popped = [4,3,5,1,2]
Output: false
Explanation: 1 cannot be popped before 2.

Constraints:

• 1 <= pushed.length <= 1000

• 0 <= pushed[i] <= 1000

• All the elements of pushed are unique.

• popped.length == pushed.length

• popped is a permutation of pushed.

Done?

Alright, remember I asked you to brainstorm and google it? So if the machine-generated solution happens to be more efficient, the argument "it's not fair because it was trained on data from the Internet" is not accepted 😉

Release the Kraken

When I initially asked ChatGPT to tackle the challenge, I copy-pasted the task from LeetCode. However, in this case, it might indeed seem like the AI pulled the solution from the depths of its "memory" without much "thinking". So let's rephrase the requirements and see if the machine can handle it. Here's the prompt I used:

The output:

Not bad, and according to LeetCode statistics, this is an average solution, more or less. Now let's push our beloved AI a bit further and ask it to enhance the code:

This is where things become really interesting! Not only did it offer an elegant solution without allocating a stack, but it also explained a potential issue that could arise from this optimization. Let's check it out on LeetCode:

The numbers speak for themselves, right? Of course, some might argue that LeetCode's benchmark is inconsistent, which is a valid point. Nevertheless, the fact remains that the AI instantly produced a decent solution and improved it in a way that not every software engineer can do.

What's Next?

So, are our jobs safe?

Yes 😅 (unless you work for Elon Musk)

To draw an analogy, today's ChatGPT (and other AI assistants like GitHub Copilot) is like an electric tool that can drill holes, drive screws, saw and more. However, if you just place it on the ground, it won't build a house for you. What is more, unskilled individuals might even harm themselves with this tool. On the other hand, experienced engineers can use the power and potential of AI to significantly improve the quality of their work.

That's it for now. Cheers!

Comments

[Anthony Fung]

Thanks for the unbiased analysis of AI.

There are far too many articles that predict the elimination of junior developer roles without exploring how future senior developers (who would supposedly use AI in their place) would be trained, or how AI would address an organisation's support needs.

[Aleksei Zagoskin]

Thanks, @ant_f_dev! It's funny how we software developers are concerned about our job security, while guys like accountants, lawyers, and many others seem unaware of what's happening. If, for instance, I were a first-level support staff member or a technical/copywriter, I'd feel somewhat stressed as well.

[Anthony Fung]

It's funny actually - fields like accountancy and law seem like areas that AI could really affect. And yet the first applications for AI being showcased focus on creative arts (writing, visual arts, music) and programming; maybe because the consequences of getting things wrong are relatively trivial? After all, if an algorithm is badly wrong, it 'just' won't compile or a developer will review and reject it.

I don't think copywriters have that much to fear at this point. From the reactions I've seen, I get the feeling that AI will greatly help them, but the output so far isn't good enough to be truly compelling marketing material. (Disclaimer: I'm not a copywriter, so I don't know how true this sentiment is)

[Aleksei Zagoskin]

It's pretty hard to guess which areas will be significantly impacted by AI first. Time will tell. In the meantime, we'll keep an eye on it 🧐

[Jeet Bhattacharjee]

This is exactly how I think about using AI for coding. It’s quite similar to how excel and other tools shrunk the market for manual calculation jobs (like statistician and book keeper). People just trained up to use the tools and be more productive, or found new niches.
Constant learning is a fundamental for a career any STEM field.

[Kevin]

I totally agree with your article. We are not endangered by the use of AI; rather, it can be seen as another tool to make your life easier (or harder if you don’t use it properly). Using ChatGPT as a day-to-day tool in software development requires a thoughtful approach. Here are some practical tips and recommendations for developers who want to incorporate ChatGPT into their workflow effectively:

1. Start with simple tasks and gradually increase the complexity

2. Trust the tool, but always use common sense and critical thinking - and don’t stop analyzing the answers it gives you

3. Use ChatGPT as a complementary tool, not a replacement for human expertise

4. Keep your skills sharp to avoid over-reliance on AI-generated code

5. Stay up-to-date on new features and functions offered by ChatGPT

I highly recommend this article from my colleague Eduardo Maciel sharing his experience using ChatGPT in a development project: scalablepath.com/machine-learning/...

[Emily Flias]

Think about GPT-5 and GPT-6, which will be released in the coming years and have trillions of parameters?
And this is baby google:
blog.google/technology/ai/code-wit...

[Aleksei Zagoskin]

True, more mind blowing things are coming!

[Rolf Streefkerk]

these are very specific computational tasks. Programming is a lot about context, to get a more accurate reading you'd need to give it a general task that it needs to solve which would require it to break them down into these kind of individual solutions.

[Kuba Wach]

Interesting experiment, thanks for sharing!