Notification type products have slowly started to rise in popularity as we get closer to the “Internet of Everything”. By the year 2025 we’ll get cars that drive you to QFC because they noticed you’re running low on milk and regularly bought it for the last 30 weeks. The correlation of you getting your milk on time leads to you performing better at work and boosting your career growth by 25% over 10 years, or so your refrigerator will report to you every week so.
But before that becomes a reality, I’ve got to adjust and write my programs by hand to give me alerts. Companies like Dataminr and IFTTT have capitalized on the notification market but in different industries. (Dataminr mainly for alerting wallstreet people, IFTTT for the common-folk). But customized scraping really has some awesome benefits to it, and this is where I think Kimono has some upside.
The NBA playoffs are coming up and my Blazers are playing the Grizzlies . Albeit if Damian Lillard doesn’t become an absolute killer and go off, my interest will probably stagnate after a first round exit when the Blazers have been 0 for 4 against them this year. That being said, I know there’s a lot of people that are interested in watching all NBA games regardless of who they’re rooting for and especially when the game is getting close. Therefore, I decided this would be a cool demonstration of a scraping mechanism by creating a notification system for when games should be close enough to watch.
NBA real time scores already have an API but I thought I would make one using Kimono’s really easy scraping chrome extension anyway. (Note: I don’t work for them, they just have a cool product). First we have to grab the values we think are important on the ESPN NBA home page. That would be the home team name, away team name, scores, and the score time. Full github code is here.
Afterwards once the API is created, all we need to do is call the API to receive a json format of the results. Since we can set the API to scrape the site every fifteen minutes, we probably get a couple hits when it’s in the fourth quarter of a game. If that’s not frequent enough, create two more APIs and run them at intervals. But NBA games usually have a lot of time-outs, especially when the games are close, so we can probably live with an update every 15 minutes.
The reason I have the values and then another nested dictionary [‘text’] in there is because each value also has a link. The team’s and score’s text on the ESPN page all have links to different pages so it creates a nested dictionary of a ‘href’ value and a ‘text’ value in case you want both.
Awesome, now that we have this, we can set up a twitter account to tweet the games when they’re getting real close! I use Tweepy to authenticate and wrap the twitter account.
You can follow the tweets right here .
IFTTT has a couple of nifty applications to automate some cross-platform devices. In it’s recipe book they have hundreds of different apps that you can post, update, tweet, etc. when another app posts, updates, or tweets. They have one for finding 2br apartments in SF but it might be nicer to customize the requirements a bit more. I don’t think I want to live in a hacker house no matter how cheap it is and definitely want to see some pictures on the postings.
I previously wrote a scraper with scrapy to parse through craigslist postings and save them into a database here. This time I’m not too interested in modifying my original code too much though and want to try out some more BeautifulSoup instead. I grabbed some of the code from Daniel Forsyth’s blog on finding tickets on craigslist. What I want to specify is any kind of place that will guarantee me under 1800 per bedroom in the SOMA or Potrero Hill district. I don’t care about the number of rooms as long as it’s within reason and the postings have to have pictures and a latitude and longitudinal location.
Cool stuff, here’s the rest of the code set up with an SMTP server to email the results directly to my email address. On the apartments module, I also tried to specify a low just in case people are putting up the 1 dollar bullshit that gets on Craigslist some of the time.
Hopefully this will inspire some people started on the right track to customize some scrapers a little more. I love scraping, it’s absolutely the best and let’s people pursue their dreams and goals. Finding cheaper housing is good. But IFTTT has done a pretty good job on craigslist. If you look more into craigslist’s filters for housing, they can actually specify pictures or not, specific titles, and neighborhoods that you might want to live in. Once you have it drilled down, all you need is an email blast for each new listing. So I like where IFTTT is heading. But I also like where Kimono is going to because they can also get into this notification marketplace too with their customized APIs. If I had the money to invest I think I would.