Python returns "Dead" when trying to read json > 240mb

Asked

Viewed 262 times

1

The problem occurs on a Centos server version 6.8 final. The same script with another smaller file works perfectly.

Test performed in Ubuntu 16.04 the process occurs successfully.

Any solution? Below is the code.

code in gist

mysql python json filing-cabinet

  • Your question is not clear enough. Clarify your specific problem or add other details to highlight exactly what you need. The way it’s written here, it’s hard to know exactly what you’re asking. See the page How to ask for help in clarifying this question.

  • Another tip is: directly paste your codes into your question. And preferably, only the snippet where the error appears. I think it’s hard for someone to stop and read and understand your 250 lines of code.

  • Thinking a little bit here, as your json is a little big, maybe while trying to read it in memory, your PC trave. Try to change the line where you read the json (jsonToPython = json.load(data_file)) to use the library as a ijson or the json-Streamer

  • 1

    Valeu... worked... https://gist.github.com/vanessasoutoc/e82a76fc7d3e45dade9ca4315c0e4b40

  • Glad you could! : ) If the answer has solved your problem, you should accept the answer to mark your question as "already answered".

1 answer

1

As his JSON eh a little big, while trying to read it in memory, your PC may crash due to lack of memory.

So try to change the line where you read the JSON:

jsonToPython = json.load(data_file)

to use a library such as ijson or the json-Streamer, which are specific for handling files with large data volume.

Browser other questions tagged mysql python json filing-cabinet

You are not signed in. Login or sign up in order to post.