How to avoid job delays and job failures

Near’s Data API, whether access through a UI or programmatically is a powerful tool that allows the export of human  movement data. Like all software, there are some best practices to follow to ensure that you can derive the maximum  possible value in the fewest steps. This article outlines some of the accumulated best practices for using the Data API. As always, contact our support team with any questions. 

Common Job Delay Reasons 

There are many reasons that a Data Export job can seem delayed. Here are the most common, with suggestions for  mitigation: 

  • Length of Analysis: Jobs can be delayed or appear to be delayed if the analysis time period is extended.  Breaking jobs up by time is a way to get data in hand faster and further analysis to begin. 
  • Company Queue: When a submitted job hits a complexity threshold any subsequent job enters a queue behind  the complex job. The complexity threshold is driven by both the count of locations (or total size) of the study  area and the type of job requested. Jobs which take significant computing time—pathing, OGS, Location  affinity, retail heartbeat are subject to queuing. To avoid queuing, consider splitting out job requests to run less  complex reports in one job and the more complex reports in a second job, allowing you to begin using the data  while the more complex job completes. 
  • Complex Shape Definition: The number of points used to define a location shape will generally slow down the  processing of determining which points are within a polygon. Mathematically, simpler shapes make for a quicker  visitor determination and quicker job finish. 
  • General API busyness: At times, a lot of companies have submitted jobs to the API. While we have a per  company queue, at times, the API can get backed up. Near generally works to increase capacity during these  times as quickly as possible, but this rare case can sometimes cause delays. 

Common Job Failure Reasons 

Geography Size 

When Data API jobs are submitted which cover a large geographical area, there is a significantly increased chance that  the job will eventually run into a failure point. This is due to the fact that the amount of raw data being processed  grows too large and outgrows the storage capability of the servers which run the Data API software. Note that in this  case, the phrase “large geographical area” can either be one large polygon or shape (The entire New York City area) or  1000s of individual polygons—such as all the McDonalds in the United States. 

In general, to ensure success for your job, follow the following guidelines: 

  • For larger analysis such as cities or high streets, break up jobs into smaller areas and then stitch the resulting files  back together. 
  • For more than ~1000 places, split your job into multiple jobs. The credit price is determined by number of  reports and places, not the number of jobs. 
  • ESRI conversion failures: if you have selected a conversion to geoDB formatted output, large jobs can sometimes  fail due to errors in the underlying open-source software used to do the conversion. Again, splitting the jobs  becomes more critical when using ESRI. 

Unsupported Reports per Country 

Not all reports are available in all countries. Jobs will fail if a report for a even just one location in unsupported country  is requested as part of the job. As of June 2022, the follow country availability is in place: 

  • Estimated Visits: supported in US, CA, AUS, NZ  
  • Extrapolated Visits: US  
  • Demographics: US and AUS. Additional countries may be available by license. 
  • Retail Heartbeat: US and CA  
  • Geosocial Affinity: US  
  • Location Affinity: US and CA  

Date Restrictions

Some reports require additional processing (for example of panels and baselines), and have restricted available date ranges: • Retail Heartbeat is available from January 1, 2018 until the day before the beginning of the previous month.  • Estimated Visits 2.0 is available from January 1, 2019 until 5 days before today. 

Any report requested outside of these timelines will generate a job failure.