I am using eBay's findItemsAdvanced API call in order to download all items from a given seller. Through my testing, I've noticed that when I hit larger sellers, say with 8000+ items, my results are iffy.
Firstly, the number of items I end up saving (using the itemID as a KEY so any duplicates are weeded out) ends up being 1-2% under the actual number reported by eBay (both on their website and through the findItemsAdvanced call). Secondly, I logged the item IDs each page I downloaded returned, and I see between 1-9 items overlapped between pages, and not only that, in both directions (that is, say page 28 overlapped page 29 by 5 items, and overlapped page 27 by 1 item). The sellers I'm working on have fixed price items and I can verify no new items were sold/removed/added during the time I initiated the download and finished it.
I download items via a threaded application, pinging eBay's API rapidly for up to 5 pages per thread (among 20 threads) all at once. I haven't been testing for any sellers of 10,000 items as I know this to be a special case/limitation of the findItemsAdvanced call. I keep the ordering as EndTimeSoonest and do 100 items per page.
What I do know is that if I were to compare the number of duplicates to the actual number of unique items, the # of duplicates + # of unique items more or less equals the number reported by eBay.
So my question is, is there a fault in my code causing this issue or a limitation of the eBay API? Is there no guarantee that each page will contain no duplicates when compared to a page before/after (depending on sorting I guess)? I realize I can download a page and try to page through the API by setting a EndTimeFrom filter based on the last item I downloaded, but this isn't achievable at the scale and speed I want to achieve. I lose the capability of threading in that scenario.
Please let me know if this is just eBay's API that I'll have to accept/work around or an issue with my idea/code.
Thank you in advanced!