超⼤JSON⽂件解析⽅案（Java）
解析超⼤JSON⽂件
1、需求
最近项⽬中需要将⼀个⼀个⼤于800M的JSON⽂件导出到Excel中，试过普通的按⾏读取⽂件和JSONReader流读取⽂件，由于JSON⽂件实在过于庞⼤，导致OOM问题
2、解决⽅案
每个json数组中包含的json对象太多，导致⽤流和按⾏读取时加载到内存会导致内存溢出。.
最终采⽤了JsonToken的解决⽅案。
dfrey.poi.util;
import com.JsonFactory;
import com.JsonParser;
import com.JsonToken;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.MappingJsonFactory;
import java.io.File;
/**
* @author godfrey
* @since 2021-12-05
*/
public class ParseJsonUtil {大文件发送
public static void main(String[] args) throws Exception {
JsonFactory f = new MappingJsonFactory();
JsonParser jp = f.createJsonParser(new File("F:/FeaturesToJSON.json"));
JsonToken current;
current = jp.nextToken();
if (current != JsonToken.START_OBJECT) {
System.out.println("Error: root should be object: quiting.");
return;
}
while (jp.nextToken() != JsonToken.END_OBJECT) {
String fieldName = jp.getCurrentName();
// move from field name to field value
current = jp.nextToken();
if ("features".equals(fieldName)) {
if (current == JsonToken.START_ARRAY) {
// For each of the records in the array
while (jp.nextToken() != JsonToken.END_ARRAY) {
// read the record into a tree model,
// this moves the parsing position to the end of it
JsonNode node = jp.readValueAsTree();
// And now we have random access to everything in the object
System.out.println("field1: " + ("field1").asText());
System.out.println("field2: " + ("field2").asText());
}
} else {
System.out.println("Error: records should be an array: skipping.");
jp.skipChildren();
}
} else {
System.out.println("Unprocessed property: " + fieldName);
jp.skipChildren();
}
}
}
}
代码中使⽤流和树模型解析的组合读取此⽂件。每个单独的记录都以树形结构读取，但⽂件永远不会完整地读⼊内存，因此JVM内存不会爆炸。最终解决了读取超⼤⽂件的问题。