I do have a hunch in that we might be able to utilize tiny LLMs to figure out parts that might possibly be brittle and combine them with traditional generation/mutation-based fuzzing to generate fuzz targets that are more likely to trigger an edge case.

I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.

// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)

[0]: https://github.com/dtcxzyw/llvm-hackme

Any fuzzer needs to compare its results to AFL (American Fuzzy Lop), "a free software fuzzer that employs genetic algorithms in order to efficiently increase code coverage of the test cases." https://en.wikipedia.org/wiki/American_Fuzzy_Lop_(software)

At one point it was considered state-of-the-art. As a project it's since been superseded by AFL++ - https://aflplus.plus/ .

> We can compare the output of the same program for both targets and flag any differences.

This is called differential fuzzing and is one of the most powerful methods to find bugs in all kinds of software.

Fantastic article. Thanks for sharing. I love the honest take on LLM-based fuzzing. It’s exactly something I would do with a similar problem.

Writing code with an LLM seems like a deal with the devil but debugging logical errors or type coercion bugs is hell.

curse that long link