What is the maximum size of a JSON file?

Asked

Viewed 3,563 times

9

I will give an example to explain my doubt:

I need to recover information from images coming from an API, this image bank returns JSON’s with information of the images I need, in all there are 33 million images and I will order 100 in 100 thousand.

The structure of a JSON if I requested 2 images would be:

{
  "meta":{
  "limit":2,
  "offset":0,
  "total_count":33640457
},
"objects":[
  {
     "artist":"sunyyasyed",
     "grid_thumb_url":"http://grid.unlistedimages.com/64830460.jpg",
     "id":64830460,
     "image_name":"k13762459",
     "preview_url":"http://comps.unlistedimages.com/ulcomp/CSP/CSP992/k13762459.jpg",
     "ratio":0.8895689,
     "resource_uri":"/api/v1.0/image/64830460/",
     "subscription_allowed":true,
     "thumbnail_url":"http://photos.unlistedimages.com/thumbs/CSP/CSP992/k13762459.jpg",
     "title":"100 Business, Web, and Office Icons",
     "type":"PHOTO"
  },
  {
     "artist":"kevron2001",
     "grid_thumb_url":"http://grid.unlistedimages.com/62322357.jpg",
     "id":62322357,
     "image_name":"k10882629",
     "preview_url":"http://comps.unlistedimages.com/ulcomp/CSP/CSP990/k10882629.jpg",
     "ratio":1.25,
     "resource_uri":"/api/v1.0/image/62322357/",
     "subscription_allowed":true,
     "thumbnail_url":"http://photos.unlistedimages.com/thumbs/CSP/CSP990/k10882629.jpg",
     "title":"Sunset with open Bible",
     "type":"PHOTO"
  }
]
}

Each image block has 13 lines, meaning at the end JSON will have ~1,300,000 lines, is there a limit to generating a JSON file? it may be that this file (with 43MB of text) may cause problems if I need to read it

json filing-cabinet

  • To say one thing, "paging" is something that goes beyond web pages, even if the term implies this, if you developed the API soon you can think of a kind of control of LIMIT via GET. After all it’s not just because life’s webapis use cute urls that invalidates the use of querystring, something like this: /api/foo/bar and /api/foo/bar?pag=2 seems to me a great output, and the rest you solve in the application by creating a "on-demand" control. Note that the LIMIT is on the server side of the API.

  • @Guilhermenascimento There is the limit parameter, so much so that the JSON itself contains the values, is that in this case as there are 33million images, it would take a long time to return small batches so I decided 100mil per batch. Ended up lasting 8 hours, were 337 lots with 100mil images each, a total of 14GB of plain text :D.

  • That’s why I mentioned one about creating a control on demand, the control would have a timeout that would add up to the current page, thus avoiding server wear and tear of your own network, of course this at the production level. But it’s just a suggestion.

  • Ahhhh, I understood the purpose of the comment :D,this API who did it was not me, is an image distributor so I do not have this control, but really they have no control , I was 8 hours direct ordering A lot information, and I wasn’t even notified

1 answer

11


In general it has no limit beyond the physical.

The problems that can cause is not knowing how to carry it in memory because it is big, Although 43MB for today is silly and does not usually take special care. If you’re using a stream then it should control access in the best way not to "engage"* the memory.

What you’re going to do with this information could be a specific problem. for example, try to read all the images in it and leave in memory.

Of course, if you use a tool that has some limit, some specific difficulty there is the problem of it and not of JSON.

*In times of electronic injection and many young people here, there are those who will not know what this is, search in the context of engine :) (is not engazopar) The term is not quite this, but it gets funnier so)

  • My fear is why all in a Streamreader and for some reason crash my back-end you know ? But for what you said 43MB is puny

  • 1

    For stream even q vc have 16MB of memory ñ will give problem in the load, then it only depends on your algorithm to treat the data in the right way.

  • Okay, thanks for the +1 guidance

  • 2

    I wanted to give an extra +1 for nostalgia.

  • @Renan did not understand the inner nostalgia of you, includes me there, explains kkk

  • *In times of electronic injection and many young people here, there are those who will not know what this is, search in the context of engine :) (is not engazopar) @Maniero dscp do not understand

  • 1

    @Leonardobonetti you are young, research :)

  • @Maniero I understood now :) is that your sentence was a little difficult to understand, but I went in the edition and understood.

Show 3 more comments

Browser other questions tagged json filing-cabinet

You are not signed in. Login or sign up in order to post.