Large JSON file processing in Java

Efficient JSON file processing is important in modern applications. If JSON file is large in size, we need to handle it properly to avoid heap out of memory error and heavy resource consumption.

GSON is one of the best library available in Java to process JSON files.

• It provides thread safe instance to reuse the same object to process multiple requests.

• It doesn't expect any annotations to be available to serialization and deserialization unless there is a special case.

• It provides toJson() and fromJson() methods for serialization and deserialization.

• It provides support to read the content using stream readers.

Below is the code snippet to read the file using java stream readers and convert to Java object. I am applying my business logic for every 100 records.

Assumption JSON file is having array of data like below.

[
{
"id": 1,
"firstName": "Emily",
"lastName": "Johnson"
}
]

Java Code

private Optional<Boolean> readFile(String path) {
    try (InputStream inputStream = Files.newInputStream(Path.of(path));
         JsonReader reader = new JsonReader(new InputStreamReader(inputStream))) {
        Instant start = Instant.now();
        reader.beginArray();
        List<User> users = new ArrayList<>();
        while (reader.hasNext()) {
            users.add(gson.fromJson(reader, User.class));
            if (users.size() >= 100) {
                //Initiate your records processing
                //You can consider processing it in async with properly handling failure scenarios 
                users.clear();
            }
        }
        reader.endArray();
        LOGGER.info("Total time taken to process file:{}, Duration:{} (millis)", path, Duration.between(start, Instant.now()).toMillis());
    } catch (Exception e) {
        LOGGER.error("", e);
        return Optional.of(false);
    }
    return Optional.of(true);
}

If you have any question please leave a question in comments section.