{"id":516,"date":"2017-12-05T16:32:16","date_gmt":"2017-12-05T21:32:16","guid":{"rendered":"https:\/\/www.mcgurrin.info\/robots\/?p=516"},"modified":"2018-05-07T23:10:41","modified_gmt":"2018-05-08T03:10:41","slug":"using-geek-power-for-good-better-living-through-code-edition","status":"publish","type":"post","link":"https:\/\/www.mcgurrin.info\/robots\/516\/","title":{"rendered":"Using Geek Power for Good: Better Living Through Code Edition"},"content":{"rendered":"
\"bottle<\/a>

Buffalo Trace bourbon<\/p><\/div>\n

Bourbon has become a hot commodity, and as it takes years to make, the supply can’t quickly ramp up to match demand. Buffalo Trace is one of many brands that have become quite popular, making it hard to find. The clerk at a local Virginia ABC store told my wife that they get a small shipment in and it “flows out like a river”, and is gone in a day. My wife found that you could check inventory of local stores online, and asked me to write a script to check it. Starting tomorrow morning, the “Buffalo Hunter” script will run once a day, check the inventory at our two closest stores, and send her a text if there are any bottles in stock.<\/p>\n

Version 1 was a fun 1-day project. I had to learn some new tricks, as the page uses client-side javascript and I hadn’t used Twilio to send texts before, but I got it all working well. Some time later I decided to port it to the cloud, using the AWS Lamda service, which had a short but steep learning curve.<\/p>\n

The website uses javascript to generate a dynamic page, so I couldn’t simply use something like Beautiful Soup to parse the html. So I used Selenium, using a Chrome headless browser on my local version, switching over to phantomjs on AWS. I switched to phantomjs because you need to have executables compiled to run under AWS, and I found a precompiled version of phantomjs on the web, and didn’t find the same for Chrome.<\/p>\n

There was one other “gotcha” I ran into. I use Windows. While I had found a correctly compiled phantomjs executable, when I zipped it along with the other files to upload, it lost its permissions settings. I could have booted up in Linux, instead I installed the Linux subsystem that’s available for Windows 10 and used bash to zip the files up. That ended up working fine. You also need to change the directory for the phantomjs log to the \/tmp\/ folder that AWS gives you write access to.<\/p>\n

In version 1, I handed off the final processed web page to Beautiful Soup because I hadn’t used Selenium’s parsing before, and I’d used Beautiful Soup’s. You can easily hand off the processed resulting web page from Selenium to Beautiful Soup (see the commented out line that starts page2Soup in the code below). When I moved to Amazon, I also figured out how to do the page scraping in Selenium, so that I didn’t need Beautiful Soup any more. The concept’s simple, but I didn’t find a good reference for the find_element)by_css_selector in python, so it took a little trial and error. .If you’re interested, here’s the version of the code that runs on AWS:<\/p>\n

Buffalo Hunter.py<\/strong><\/p>\n

\r\nimport logging\r\nimport datetime\r\nimport time\r\n# from bs4 import BeautifulSoup\r\nfrom selenium import webdriver\r\nfrom twilio.rest import Client\r\n\r\naccountSID='SID Here'\r\nauthToken = 'token here'\r\nstores = {'219': 'Old Courthouse' , '231': 'Maple Ave.'}\r\n\r\n# options = webdriver.ChromeOptions()\r\n# options.add_argument('headless')\r\n# driver = webdriver.Chrome('c:\/program files (x86)\/chromedriver.exe')\r\ndriver = webdriver.PhantomJS(executable_path=\"\/var\/task\/phantomjs\", service_log_path='\/tmp\/ghostdriver.log')\r\n\r\ndef myhandler(event, context):\r\n\ttry:\r\n\t\tresults = ''\r\n\t\tsuccess = 0\r\n\t\tfor store in stores:\r\n\t\t\tdriver.get('https:\/\/www.abc.virginia.gov\/stores\/'+store)\r\n\t\t\tmake_my_store = driver.find_element_by_id('make-this-my-store')\r\n\t\t\tmake_my_store.click()\r\n\t\t\ttime.sleep(5)\r\n\t\t\tdriver.get('https:\/\/www.abc.virginia.gov\/products\/bourbon\/buffalo-trace-bourbon#\/product?productSize=0')\r\n\t\t\ttime.sleep(5)\r\n\t\t\telement = driver.find_element_by_css_selector('td[data-title=\"Inventory\"]')\r\n\t\t\t# page2Soup = BeautifulSoup(driver.page_source, 'lxml')\r\n\t\t\t# element = page2Soup.find(\"td\", {\"data-title\": \"Inventory\"})\r\n\t\t\tinventory_value = element.text\r\n\t\t\tif inventory_value <> '0': success = 1\r\n\t\t\tresults= results+stores[store] +' has '+inventory_value+ ' bottles of Buffalo Trace. '\r\n\t\tdriver.close()\r\n\t\tdriver.quit()\r\n\t# Send results if inventory not 0 at both stores\r\n\t\tif success == 1:\r\n\t\t\tresults = 'Success! ' + results\r\n\t\t\ttwilioCli = Client(accountSID, authToken)\r\n\t\t\tmyTwilioNumber = 'myPhoneNumberHere'\r\n\t\t\tdestinationCellNumber = 'destinationCellNumberHere'\r\n\t\t\tmessage = twilioCli.messages.create(body=results,from_=myTwilioNumber, to=destinationCellNumber)\r\n\texcept Exception as e:\r\n\t\tlogging.error(str(datetime.datetime.now())+' Error at %s', 'division', exc_info=e)\r\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
Bourbon has become a hot commodity, and as it takes years to make, the supply can’t quickly ramp up to match demand. Buffalo Trace is one of many brands that have become quite popular, making it hard to find. The … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[75,74,12,40,73,69],"_links":{"self":[{"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/posts\/516"}],"collection":[{"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/comments?post=516"}],"version-history":[{"count":6,"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/posts\/516\/revisions"}],"predecessor-version":[{"id":551,"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/posts\/516\/revisions\/551"}],"wp:attachment":[{"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/media?parent=516"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/categories?post=516"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mcgurrin.info\/robots\/wp-json\/wp\/v2\/tags?post=516"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}