pythonComputation for the Social Sciences

Introduction to Computation for the Social Sciences Assignment 7

Prof. Dr. Karsten Donnay, Stefan Scholz Winter Term 2018 / 2019

Please solve the exercises below and submit your solutions to our GitHub Classroom until Dec, 18th midnight. Submit all your code in one executable file (py

/ ipynb ) and your list in one delimited text file (csv).

This assignment sheet is graded according to the following scheme: 3 points (fully complete and correct), 2 points (mostly ), 1 point (partly ) and 0 points (hardly). You will get individual feedback in your repository.

Exercise 1: Simple Web Crawler

Web Crawling, i.e. the automated access of Web resources via software, and Web scraping, i.e. the extraction of content from Web resources, are essential techniques to gather data for many analysis tasks in the social sciences. We will implement a very simple Web crawler that starts at a given URL and iteratively accesses all links, more specifically all href-tags in the HTML source code of a Website.

To get an idea how to approach this problem using Python, have a look at the sample implementation of a crawler in the subchapters Traversing a Single Domain and Crawling an Entire Site in the book Web Scraping with Python by Ryan Mitchell. If you search for the book online, you will find a PDF version of it.

Implement an iterative crawler (no recursive calls) that:

  • opens the front page of the German Wikipedia1 and downloads the html resource

  • parses the html file as a BeautifulSoup object.

  • collects all internal href tags, i.e. only links to other Wikipedia pages on the server de.wikipedia.org

  • opens each of the collected links and parses the returned html resource for additional internal links

    Additionally, your crawler has to meet the following requirements:

  • Consider the Wikipedia front page as the root (level 0) of a tree and the Wikipedia pages linked to on the front page as level 1. Crawl no deeper than level 2!

  • Visit each unique link only once, i.e. disregard links that you traversed before

  • Store all links of the same level in a list, i.e. one list per level.

  • Include links that you encounter more than once in the list of the lowest level (closer to the root) they appear at.

  • Write the lists of links to a CSV file in the format level ; link_URL.

Finally, submit your solutions by adding your code and csv file in your Git repository within the folder assignment07 > solution. Do not forget to commit and push your solutions afterwards.

1 https://de.wikipedia.org/wiki/Wikipedia:Hauptseite

相关推荐
欢乐马87914 分钟前
优化|MPSolver 和 pywraplp 到底谁是谁?一次讲清 OR-Tools 的三层结构
python
mengge.cloud1 小时前
Python基础保姆级!从入门到入土
开发语言·python·云原生·pygame
qq_158895411116 小时前
Suno-API 是什么?官方 V6 模型与 Python / TypeScript 接入指南
python·ai音乐·suno api
我不会插花弄玉12 小时前
13.多路转接epoll【由浅入深-网络】
网络·c++·epoll
郝学胜-神的一滴14 小时前
Effective STL 条款03:容器中的对象复制要廉价且正确
开发语言·数据结构·c++·算法·rpc·stl
西西弗Sisyphus15 小时前
C++ explicit 关键字工程实践
开发语言·c++
学无止境小袁同学16 小时前
Google ADK 学习指南
python·llm·agent·多agent·google adk·llmagent·agent cli
郝学胜-神的一滴17 小时前
Effective Python 条款35:用类封装替代生成器异常注入
开发语言·人工智能·python·金融·pandas·scikit-learn·matplotlib
LK_0717 小时前
【力扣】142. 环形链表 II |Python
python·leetcode·链表
wuyk55517 小时前
Python办公自动化-第02讲-Excel文件认知与四大库选型
开发语言·python·excel