I have often discussed code generation with large language models (LLMs) and have shared a lot of experience, including the pitfalls and best practices from more than 60 AI projects, including the lack of modularity and error handling.
There are numerous tools for quality control, but sometimes it is more appealing to create a smaller “brain” that checks code according to a set of human-language policies without writing any code. The aim would be to fine-tune a small model (for instance, llama3) to double-check any code generated by other models (including larger ones that might become tired after a long conversation and start producing low-quality code).
In More Detail
- Fine-tune a small and cheap model to review code generated by other models, whether small or large. This is done according to a set of policies that we define. We will supply a code snippet to the model, and it will respond with a score for each criterion. Depending on the model's response, we can either accept or modify the code under review.
- Fine-tuning and inference for less than $3. The exact pricing can be found on fireworks.ai. You could alternatively use any other framework like Hugging Face and its auto-training features. However, I personally like Fireworks because it is easy to train and deploy, and I find the inference to be fast. Of course, you can choose whatever suits your use case.
A sample response should look like this (in order to be able to be processed by another tool such as a CI/CDI pipeline).
```{
"modularity_score": "7",
"modularity_description": "The code is well-organized into independent sections, such as the HTML structure, CSS styles, and content.....",
"error_handling_score": "5",
"error_handling_description": "The code does not include any error handling mechanisms. ...",
"logging_score": "6",
"logging_description": "The code does not include any logging statements. While it's effectively.",
"explanation": "The code is well-structured and follows standard HTML and CSS practices. However, it lacks modularity in the CS.... code."
}```
Top comments (0)