Web Scraping -101

Web scraping

Now, we know that we need web scraping, let us start with web scraping. Today we will scrape very simple data using a very beginner friendly python library.
Today, we will be needing two python libraries at first let us install them using pip and then import them.

Libraries

Scraping

Here I am installing them together, you can install them one by one
pip install requests
pip install bs4

For me it is already installed so it is showing ‘already satisfied’. Your might take a few seconds.

Scraping

And then we save the source code in a variable and then use the bs4 library to parse and extract the required data.

We used BeautifulSoup, which is a class in the bs4 library and it takes the source_code which we got as a response from the website.

Now we will use selector to find out the data we seek.

Let us say we will build a program which will extract the number of Covid-19 case from the worldometers.info website. for that we need to get the url to the page where we will get the data we want from and then use the requests library to send a HTTP request and the website will give us the source code.

Scraping

Parsing the data

Scraping

we can get it easily from the title and then split it to get the number.

Scraping
recovered

Now let us say we are interested in the number of recovered cases which we can not scrape from title. So let us look for the data in the HTML source code or we can inspect the element from the browser and get the tag or class for the element.

We can see the element has a ‘span’ tag and a ‘maincounter-number’ class. which we can use to extract the data.

This is bad, it has spited the number of total cases. Because from the source code we can see those both elements the same class and tag name. So the program has just gave us the first value it encountered.

So, Now you are familiar with scraping, it is your job to find out how you can scrape the number of the recovered cases. And that way you can scrape most of the websites. Thank you. Happy scraping!

41 thoughts on “Web Scraping -101”

  1. I truly love your site.. Great colors & theme. Did you develop this site yourself?
    Please reply back as I’m trying to create my own site and would like to know where
    you got this from or just what the theme is named. Thank you!

  2. I’m really inspired along with your writing
    skills as well as with the structure in your blog. Is that this a paid
    topic or did you customize it yourself? Anyway stay up the excellent quality writing, it is uncommon to see a great
    weblog like this one nowadays..

  3. I’m more than happy to discover this website.
    I wanted to thank you for your time for this particularly wonderful read!!
    I definitely loved every little bit of it and i
    also have you book marked to see new things in your site.

  4. It’s actually a cool and helpful piece of information. I am happy that you simply shared this useful info with
    us. Please stay us up to date like this. Thank you for sharing.

  5. If some one needs to be updated with newest technologies afterward he must
    be pay a quick visit this website and be up to date every day.
    ps4 games 185413490784 ps4 games

  6. I know this web page provides quality depending posts and additional data, is there any other site which
    gives these kinds of data in quality? ps4 games allenferguson ps4 games

  7. Aw, this was an extremely good post. Taking a few minutes
    and actual effort to make a really good article…
    but what can I say… I hesitate a lot and don’t seem
    to get anything done.

  8. W88 Casino Review
    Established in 2013, W88 has been a leading online gaming site in Asia for quite
    a while. During this time, the operator has earned a great reputation among players thanks to its wide range of activities, which include casino games, sports betting, lottery,
    live casino games and poker.

    The W88 site is well-known for its commitment to customer service, offering players the
    chance to claim generous bonuses regularly. So, it while it seems like this casino is packed with great games and customer service, let’s find out if there’s more to it than this.

  9. Hey there I am so delighted I found your weblog, I really found you by mistake, while I was
    looking on Askjeeve for something else, Nonetheless I am
    here now and would just like to say thanks a lot for a marvelous post and a all round interesting blog (I also
    love the theme/design), I don’t have time to go through it
    all at the moment but I have book-marked it and also included
    your RSS feeds, so when I have time I will be back to read much more, Please do keep up the superb work.

  10. This is a good tip especially to those new to the blogosphere.
    Short but very precise information… Appreciate your
    sharing this one. A must read article! asmr 0mniartist

  11. Good day! I know this is kind of off topic but I was wondering which blog platform are you using for this site?
    I’m getting fed up of WordPress because I’ve had problems with hackers and I’m looking at alternatives for another platform.
    I would be fantastic if you could point me in the direction of a good
    platform. asmr 0mniartist

  12. Hi there this is kinda of off topic but I was
    wanting to know if blogs use WYSIWYG editors or
    if you have to manually code with HTML. I’m starting a blog soon but have
    no coding expertise so I wanted to get advice from
    someone with experience. Any help would be greatly appreciated!
    0mniartist asmr

  13. each time i used to read smaller articles or reviews which also clear their motive, and that is also happening with
    this paragraph which I am reading here. asmr 0mniartist

  14. Terrific work! That is the type of info that are supposed
    to be shared around the internet. Disgrace on Google for no longer positioning this post higher!
    Come on over and seek advice from my web site
    . Thank you =)

  15. When I originally commented I clicked the “Notify me when new comments are added” checkbox
    and now each time a comment is added I get four emails with the same comment.
    Is there any way you can remove people from that service?
    Thank you!

  16. Hi there, I discovered your site by the use of Google even as searching for a similar topic, your web site came up, it
    appears to be like great. I’ve bookmarked it
    in my google bookmarks.
    Hi there, simply turned into aware of your weblog
    through Google, and located that it is truly informative.
    I am gonna watch out for brussels. I will be grateful for those who proceed this in future.
    Lots of other people shall be benefited from your writing.
    Cheers!

  17. I absolutely love your blog.. Very nice colors & theme.
    Did you develop this amazing site yourself?

    Please reply back as I’m looking to create my own personal site and want to know where you got this from or exactly what the theme is named.
    Appreciate it!

  18. Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates.
    I’ve been looking for a plug-in like this for quite some time and was hoping maybe you
    would have some experience with something like this.
    Please let me know if you run into anything.
    I truly enjoy reading your blog and I look forward to your new updates.

  19. scoliosis
    Aw, this was an exceptionally nice post. Taking the time and actual effort to generate
    a good article… but what can I say… I hesitate a lot and don’t seem to get anything done.
    scoliosis

  20. Pingback: Google

  21. Hi it’s me, I am also visiting this web site on a regular basis, this site is genuinely good and the viewers
    are genuinely sharing fastidious thoughts.

  22. I used to be recommended this website by way of my cousin. I am no longer sure
    whether or not this post is written by him as nobody else understand such specific about my difficulty.
    You’re incredible! Thank you!

  23. Pingback: Google

  24. you’re actually a good webmaster. The web site loading pace is
    amazing. It seems that you’re doing any unique trick. Furthermore,
    The contents are masterpiece. you’ve performed a magnificent
    job in this matter!

  25. Hello, I believe your site could possibly be having browser compatibility issues.
    When I take a look at your website in Safari, it looks fine but when opening in Internet Explorer, it’s got some overlapping issues.
    I just wanted to provide you with a quick heads up! Apart from that,
    excellent blog!

Leave a Comment

Your email address will not be published.