Stefan Bielmeier

<< articles & projects

Airbnb Wifi Crawler

August 18, 2021

Working remotely at AirBnBs has one bottleneck

I went to Italy to an Airbnb with 4 friends to do workation in the summer of 2020. Workation means working 4 days from somewhere else, and spending the long weekend. The airbnb said it had Wifi. I trusted it. We went.

Unfortunately, the Wifi turned out to be a mobile network box with a SIM card connected 4G box with terrible cell reception. Of course, we couldn’t properly do zoom calls for work, the battery of the thing went out every 4 hours, and after 10 GB after day 2 we ran out of data. Another problem was we couldn't switch to using our cellphones as a hotspot because cellreception generally was bad. So: woof. Never again.

For my next "workation" in summer of 2021, I picked a place (lake como in Italy) and messaged 50 hosts manually and asked them for their wifi speeds. It was nice to have information about the network, speeds, and reliability, but it was a pain to spend the time and write all those messages.

So then I went on and built a crawler for Airbnb wifi speeds to automatically give me the wifi information. Given a airbnb search URL, it will send messages to the first (e.g. 50) hosts for that search query. The messages ask how fast the download and upload speeds are, whether there are any speed or data limits (common in Italy, for example), and what type of connection it is (cable, phone line, cell network).

After, a cron job will check the inbox daily for any new messages. The cron job also enters the host’s responses into a google spread sheet with the associated airbnb room id and a link to the room. This is a snippet of that list:

List

Afterwards, a bunch of my friends asked me to check AirBnBs for them through links in the summer of 2021. It was nice to see them ask for help on this. I had my first users! With a list of about 100 Airbnbs with host-verified Wifi information, I thought about launching this project on a bigger scale for other people. But, after speaking with my friends who built a startup in the web scraping business, I concluded that it was probably not a super good idea to do that.

The solution probably doesn't align with Airbnb's policies and heavily depends on access to their platform. You can't just send messages to people with an Airbnb account to get information like that. This extensive depencency limits the project's potential significantly. This is why I chose to discontinue the project (edit: that was before Airbnb launched their verified Wifi speeds feature in November 2021).

Still, it was a fun to work on this. Especially with the three learnings I made on the way:

A) Large companies do not seem to change their log-in flow or layout of the application all that much after some time. In terms of DOM, it becomes rather static! My crawler did not break in a period of 5 months, which I found quite suprising.

B) It is interestingly hard to is to build a code-based puppet for a complicated web application. Especially identifying the correct elements in the DOM, involve lots of trial and error.

C) Solving your own problems to get users is not enough. Even though it works, it may be useful to think about risks and dependencies of the solution before investing resources into building it.

For reference, check out the (remains) of the project here

Edit Feb 2022:

I feel like the Airbnb Internet problem still exists. In January, my friend Felix and I went to Canary Islands for a couple of days to get some sun and work on projects. Interestingly, the first thing I found myself doing as a "work from anywhere" person was choosing the destination. We chose to go to Canary Islands because it's pretty much the warmest place in the (continental) EU. After that, I checked out a bunch of potential Airbnbs with a good price, nice location in a city close to the beach, and good amendities / pretty apartment. Only after making that shortlist I checked if these apartments had fast internet.

Luckily, in Canary Islands, the average Internet speed was good. I only had to message 3-4 hosts how fast the internet was at their places. Most hosts could offer at least 30/10 Mbps Download and Upload. So we quickly came to a conclusion and chose one. Reflecting on this a bit, I made a bunch of observations.

One, there is a process how I pick Airbnbs to work from. I would choose the destination first and primarily. So if I want to go to Italy, or Lisbon, or Canary Islands, I would chose that destination for the weather or because I haven't visited it yet. Or because it seems exciting to some degree. Then, after picking a region or city, I make a preselection of places that I like. Sufficient criteria for me are the combination of good price, good location in the city, and a pretty apartment with good amendities. Only then I ask the hosts about internet speeds of the apartments.

Second, that process reveals that the pain level of finding an Airbnb with good internet scales with bad average internet speeds in a region. So in Canary Islands, I got lucky – only had to message 3-4 hosts. Not a problem at all and took me 1h because the average internet is good. But for Lake Como, I messaged 50 potential hosts because the internet on average is quite terrible there, and I just had to message more to find a 1) nice apartment with 2) good internet. And that was quite painful because it took a bunch of hours of back and forth because the hosts also didn't send complete information. I couldn't book a place until I knew if the internet worked because otherwise I'd have to rely on my phone's 4G only which I didn't want to do in a region with bad internet.

Third, I was surprised the problem still exists after Airbnb introduced their verified internet speed feature. With this feature, hosts can display the internet speeds with their listing on the platform. Intuitively I was assuming the stubborn minority of "remote workers" would simply get hosts to add that information to their listing after hosts got annoyed with these requests. But even in a place like Las Palmas, which is quite popular for remote work (see NomadList), that isn't the case. Most listings didn't have that internet speed information. Maybe the hosts will have their place booked anyway, and don't really care about adding that information. Or Airbnb hasn't launched it in Spain yet. I'm not sure.

So finally if I look at this internet problem from a business perspective: I mean yes, I probably work from somewhere else once or twice a year for a week or two, and a lot of other people do as well. But the pain of writing a bunch of hosts with nice apartments (which I pre-selected) how fast the internet is is honestly not that bad for places that have good internet. And seem to be about half of places you can go.

And for places that on average have a lot worse internet speeds, you'll just have to contact a more hosts. You do it once, and then you're done. It takes about an hour to send messages to all hosts who have apartments you like, and once you find one, you're done. So it'll also not take that long. It's not that frequent of a problem. But I really had to start somewhere, I would start with amazing locations with bad internet speeds to find the outlier Airbnbs with good internet. They exist. On a grander scale however, if I had to pick a big problem to solve, this probably wouldn't be the one – probably not a problem that's big enough. And in combination with the legal thingy, on to looking for bigger problems...

Did I miss something? Let me know what you think.