Commit Graph

Select branches

Hide Pull Requests

cat_loop_page_mutli

curate_by_pd

master

Mono Color

• 263512ae7d added ML training projects in pytorch master scott 2022-08-02 20:14:38 -07:00
• 597372cad3 needless files removed spbeach46 2022-02-24 21:21:22 -07:00
• 0bec7088c9 .gitignore updated again spbeach46 2022-02-23 17:29:18 -07:00
• bed9d4531c .gitignore update spbeach46 2022-02-23 17:08:13 -07:00
• 893e936ff3 added loop revision for all listings spbeach46 2022-01-28 18:05:54 -07:00
• faa639ea83 adding random revision functionality to show account/store activity spbeach46 2022-01-27 21:00:13 -07:00
• 2e10be4469 included option for variable range of images per listing scott 2022-01-14 19:53:59 -07:00
• 19723c0ea5 fixed oauth renewal errors. added functionality for retrieving list of mens and womens cats separately scott 2022-01-12 22:25:40 -07:00
• 6a3c10fdef added oauth token renewal for shopping api scott 2022-01-11 16:27:57 -07:00
• 3ee619ae3c allow dict_pics to write dict_pics.txt scott 2022-01-09 19:30:23 -07:00
• a4f93af3ee dict_pics cleanup and changed tag file extensions to lowercase scott 2022-01-09 19:16:56 -07:00
• 8c41fe1daf new dl_pic, dict_pics methods and updated dl_pictures. fixed bugs scott 2022-01-07 18:28:37 -07:00
• ae25ab88b6 dl_pictures and curate.py cleanup scott 2022-01-07 02:47:22 -07:00
• e879e80910 blah scott 2022-01-04 20:53:34 -07:00
• fc793f4456 get_item_fromfindItemsByCateogry fix. Invalid token errors persist scott 2022-01-04 20:50:42 -07:00
• 7d8be32ba8 commit before checkout 428e0379ef for ref scott 2022-01-03 13:32:42 -07:00
• 525f46cc34 commit before checkout 428e0379ef for refs scott 2022-01-03 13:31:19 -07:00
• 6650468756 replaced FindingApi for easy refining of cat search scott 2021-12-31 15:30:06 -07:00
• 5ec46ae0c7 commit before replacing FindingApi scott 2021-12-31 15:08:48 -07:00
• b349d2f07a added separate dict_pics function for changing target training images url on the fly spbeach46 2021-12-17 22:26:55 -07:00
• 2f3df22c4a added image_faults.py 2 remove faulty images, fixed non-expand PictureURL dfs scott 2021-12-12 19:22:09 -07:00
• e766a1ba9d updating update_cats for oauth scott 2021-12-07 13:46:24 -07:00
• 29800af1d4 unique ids fix scott 2021-12-01 13:43:14 -07:00
• 93297f952e change master cat_loop_page_mutli scott 2021-11-30 18:32:01 -07:00
• 97ab3c7dab saves list of 20_itemids instead of list of ids scott 2021-11-30 12:43:37 -07:00
• 61e6770812 Removed unused FindingApi class/methods scott 2021-11-30 12:32:34 -07:00
• b6796a789e get_item_from_findItemsByCategory returns None if call limit is reach. conky will save and exit scott 2021-11-28 01:19:59 -07:00
• 5a6963b371 ip limit fix scott 2021-11-28 00:17:34 -07:00
• eaf3480341 added random sleeps before making shopping api call scott 2021-11-26 22:38:01 -07:00
• 2cf8270707 get_item_from_findItemsByCategory connection error fix scott 2021-11-26 20:40:26 -07:00
• 0f65ab0bc2 scrape_ids working. Added function to conky in ebay_api scott 2021-11-26 18:46:51 -07:00
• ab064ce5cf traditional id scraper with BeautifulSoup spbeach46 2021-11-25 18:31:54 -07:00
• eb780faf40 modTimeFrom w dateutil datetime timedelta parser integration spbeach46 2021-11-21 16:05:20 -07:00
• 7cee062b78 threaded cat loop page attempt 2 with modTimeFrom spbeach46 2021-11-21 12:47:04 -07:00
• 1faa4e86fd duplicate id fixes. full data not getting fetched scott 2021-11-15 21:01:02 -07:00
• 6bdc94c8c6 fixing call limit errors and duplicate ids scott 2021-11-10 18:01:09 -07:00
• 428e0379ef comm before finding edit scott 2021-11-10 15:37:26 -07:00
• 225cf91f7b commit before merge master curate_by_pd spbeach46 2021-06-10 12:28:41 -07:00
• 2c5b878977 commit before push spbeach46 2021-06-10 01:53:36 -07:00
• fc7449827c added option to expand pics or not. Meaning @ dl only primary pic will dl spbeach46 2021-06-02 00:29:58 -07:00
• f72bfe38cb added HideDuplicate and sortOrder params to Findin class/ finding api call spbeach46 2021-06-01 23:12:55 -07:00
• 5582cd29ef working spbeach46 2021-06-01 22:28:54 -07:00
• 9fc00f4eab dropped na values @ training df. Used reindex instead of loc to select columns spbeach46 2021-05-30 15:51:05 -07:00
• c2a812f7a8 added used items query as default paramter in get_data() method spbeach46 2021-05-27 22:00:27 -07:00
• 152b72f5df added dict for service and page input options spbeach46 2021-05-25 18:47:23 -07:00
• 35cdf8374f adding csv file save and appending functionality where needed spbeach46 2021-05-11 11:11:59 -07:00
• c8ab1b13d9 Added method to update category IDs var and txt file spbeach46 2021-05-10 15:14:57 -07:00
• 6cfa798902 Fixing more bugs in dl_pictures spbeach46 2021-04-27 22:42:55 -07:00
• 545bc78421 fixed rest of dl_pictures bugs. Needs a little more testing spbeach46 2021-04-24 15:23:47 -07:00
• 6ccce53b10 fixed some file handling bugs with dict_pics.txt spbeach46 2021-04-21 17:12:35 -07:00
• 2f99c63165 fixed more dl_pictures bugs spbeach46 2021-04-20 21:09:35 -07:00
• 5532a82d8d debugging dl_pictures spbeach46 2021-04-15 18:26:42 -07:00
• 34a6451400 mostly finished dl_pictures(). Need testing spbeach46 2021-04-13 10:10:24 -07:00
• fb327a9375 changing custom dict to dl_pictures function and making temp_dict_pics.txt file with target_dirs.txt spbeach46 2021-04-09 22:37:14 -07:00
• ef237b8a1b added dict_pics.txt updating for expanded_dfs method spbeach46 2021-04-07 15:50:23 -07:00
• 622b824eaf fixed dict_pics errors. made more robust regex pattern spbeach46 2021-04-05 14:38:07 -07:00
• fcbe4e088c added dict_pics output to curate.py and set destination var in ebay_api.py spbeach46 2021-04-04 14:45:56 -07:00
• b205b44b50 added function to create custom image url dictionary spbeach46 2021-04-04 14:38:04 -07:00
• 35100b7952 fixed extract_contents for expanded_dropd df spbeach46 2021-04-03 13:09:21 -07:00
• 167f1f29ec added download fix spbeach46 2021-04-02 23:42:31 -07:00
• 3c16301a96 Created curate file for running curate methods from ebay_api.py spbeach46 2021-04-02 11:08:56 -07:00
• 6abf695908 added some exceptions but still likely incomplete spbeach46 2021-02-14 01:34:56 -07:00
• 816fb797fa added function to extract list values from cells spbeach46 2021-02-08 20:16:58 -07:00
• 855e65af80 dropping cols from nvl_df and combining class_training with nvl_training spbeach46 2021-02-07 12:25:37 -07:00
• 1087027812 corrected for setcopywarning spbeach46 2021-01-31 22:13:52 -07:00
• fc0090ea3e added config file import spbeach46 2021-01-31 21:49:24 -07:00
• 627992036a reverted back to creds in script. change to config file method spbeach46 2021-01-30 03:38:26 -07:00
• 4f46741a0f added notes and changed security_appname to var spbeach46 2021-01-30 03:08:10 -07:00
• 72db161858 created def for class training set and nvl training set. Still unfinished spbeach46 2021-01-29 23:48:19 -07:00
• 18871a76d7 added nvl_dict function to create series based on nvl dict of raw data spbeach46 2021-01-26 23:01:00 -07:00
• d09090dc13 deleted curate defs that work from json directly instead of pandas df spbeach46 2021-01-24 22:46:55 -07:00
• 92516ff4f0 adding gameplan for constructing two csv file from pandas after shopping api call; for main data and training set spbeach46 2021-01-23 01:38:04 -07:00
• fd6fd04ecd attempting curate by starting with pandas df first spbeach46 2021-01-22 23:21:56 -07:00
• b4e1961ace working on curate_data to remove formatting issues in csv spbeach46 2020-12-29 01:55:21 -07:00
• dcbdb7ced3 need to fix data.update in conky to be list of dicts not dict of dicts spbeach46 2020-12-29 00:20:55 -07:00
• e6cf1b6a5d fixed repeats and multithreading. Needs refining spbeach46 2020-12-27 21:13:12 -07:00
• 7af5811792 added lambda function to multithreader in FindingApi method spbeach46 2020-12-25 12:15:20 -07:00
• d9781579cd wtf spbeach46 2020-12-12 18:56:51 -07:00
• bcb11de855 changed data var returned in shoppingapi to data['Item']. Vice versa in update_data func. This is so data.update is correct spbeach46 2020-11-12 14:46:37 -07:00
• 97c5900a5b added multithreading to both finding and shopping apis including conky() def spbeach46 2020-11-12 13:22:51 -07:00
• 1134d3f155 adding multithreading to first call in main spbeach46 2020-11-08 18:47:03 -07:00
• 5965f19d2a adding functions to curateData class spbeach46 2020-11-07 10:39:49 -07:00
• cd21c98e54 removed Unbound assignment error for accessing training.csv during exception handling. working on runtimeerror for mutable dictionary in curate_data loop spbeach46 2020-11-05 15:32:41 -07:00
• 904014ea38 added conditional to check if item id in both item_id_results_list and in local file to eliminate variations listings including written data spbeach46 2020-10-30 21:55:40 -07:00
• 47b77600a5 corrected update_df to iterate over items and nvl kv pairs. and began check for repeat results spbeach46 2020-10-24 03:36:31 -07:00
• b64e2f74c1 had to convert inputs to ints and correct filenotfounderror in get_ids_from_cats fuction spbeach46 2020-10-18 15:32:17 -07:00
• 26b425f31c Changed to_csv mode to append to allow for writing if not present and appending if present spbeach46 2020-10-18 13:56:16 -07:00
• 60b8f8979c changed big_data names to training spbeach46 2020-10-18 00:08:04 -07:00
• 7c338e69d5 added error handling for initial/empty csv file spbeach46 2020-10-17 17:22:45 -07:00
• ae5a4e92bb added CurateData class for preprocessing pipeline fucntionality spbeach46 2020-10-17 16:21:11 -07:00
• 58d8c8cda7 added get_data method call to get_ids_from_cats call spbeach46 2020-10-12 18:42:57 -07:00
• 68f1341012 added get_data method for easier debugging. Added zeroth index term to ...['itemId'] spbeach46 2020-10-12 17:55:07 -07:00
• 12f4770eaf corrected pageNumber and service params. Need to fix typeError on line 24-27 spbeach46 2020-10-12 11:48:15 -07:00
• acb5955b97 comment before adding pagenumber and service to init spbeach46 2020-10-12 00:53:29 -07:00
• aa73991c3c fixed classes, variables, methods to not show any terrible errors spbeach46 2020-10-11 23:11:59 -07:00
• 47ad8639ff changing defs to methods and variables to attributes. Trial and error. Code seems to be broken right now spbeach46 2020-10-11 22:12:01 -07:00
• 9d02381140 completed conditional to check for itemId existence in csv data and 20-itemId string list generator spbeach46 2020-10-08 03:07:41 -07:00
• 873050f898 removed list(map(str,item_id_results)) because itemIds are already strings spbeach46 2020-10-08 00:20:01 -07:00
• 3b0b96fd92 around line 37, creating itemId list to pipe into shopping api call/class spbeach46 2020-10-07 18:53:19 -07:00
• b5afa5cfc0 removed deleted stuff? spbeach46 2020-10-04 23:48:52 -07:00