From f40825139b9c698c07facc7d77e57ee3b58fad55 Mon Sep 17 00:00:00 2001 From: spbeach46 Date: Thu, 11 Jun 2020 17:54:12 -0700 Subject: [PATCH] added workthrough comments at bottom of page --- posh.py | 1 + 1 file changed, 1 insertion(+) diff --git a/posh.py b/posh.py index 1209c07..84bd9ca 100644 --- a/posh.py +++ b/posh.py @@ -117,3 +117,4 @@ def main(): print(total_results) main() +'''to speed up the program you can include a few things: 1) only parse the total results and sift for the NWT listings to create a separate NWT list 2) Implement processpoolexecutor to use more than one worker to parse the pages 3) find a better way to find the last page so you don't have to make more requests than necessary. This could be either taking the "smallest" "no listings found" page of the pages while excluding the others after the smallest one is found. Or, determining from the request headers whether a page is worth downloading or not 4) using a while loop in chunks of 2-4 pages to find the last page in conjunction with number 3''' \ No newline at end of file