pythonComputation for the Social Sciences

Introduction to Computation for the Social Sciences Assignment 7

Prof. Dr. Karsten Donnay, Stefan Scholz Winter Term 2018 / 2019

Please solve the exercises below and submit your solutions to our GitHub Classroom until Dec, 18th midnight. Submit all your code in one executable file (py

/ ipynb ) and your list in one delimited text file (csv).

This assignment sheet is graded according to the following scheme: 3 points (fully complete and correct), 2 points (mostly ), 1 point (partly ) and 0 points (hardly). You will get individual feedback in your repository.

Exercise 1: Simple Web Crawler

Web Crawling, i.e. the automated access of Web resources via software, and Web scraping, i.e. the extraction of content from Web resources, are essential techniques to gather data for many analysis tasks in the social sciences. We will implement a very simple Web crawler that starts at a given URL and iteratively accesses all links, more specifically all href-tags in the HTML source code of a Website.

To get an idea how to approach this problem using Python, have a look at the sample implementation of a crawler in the subchapters Traversing a Single Domain and Crawling an Entire Site in the book Web Scraping with Python by Ryan Mitchell. If you search for the book online, you will find a PDF version of it.

Implement an iterative crawler (no recursive calls) that:

  • opens the front page of the German Wikipedia[1] and downloads the html resource

  • parses the html file as a BeautifulSoup object.

  • collects all internal href tags, i.e. only links to other Wikipedia pages on the server de.wikipedia.org

  • opens each of the collected links and parses the returned html resource for additional internal links

    Additionally, your crawler has to meet the following requirements:

  • Consider the Wikipedia front page as the root (level 0) of a tree and the Wikipedia pages linked to on the front page as level 1. Crawl no deeper than level 2!

  • Visit each unique link only once, i.e. disregard links that you traversed before

  • Store all links of the same level in a list, i.e. one list per level.

  • Include links that you encounter more than once in the list of the lowest level (closer to the root) they appear at.

  • Write the lists of links to a CSV file in the format [level ; link_URL].

Finally, submit your solutions by adding your code and csv file in your Git repository within the folder assignment07 > solution. Do not forget to commit and push your solutions afterwards.

1\] https://de.wikipedia.org/wiki/Wikipedia:Hauptseite

相关推荐
序属秋秋秋1 小时前
《C++初阶之内存管理》【内存分布 + operator new/delete + 定位new】
开发语言·c++·笔记·学习
木头左2 小时前
逻辑回归的Python实现与优化
python·算法·逻辑回归
quant_19863 小时前
R语言如何接入实时行情接口
开发语言·经验分享·笔记·python·websocket·金融·r语言
失败又激情的man8 小时前
python之requests库解析
开发语言·爬虫·python
打酱油的;8 小时前
爬虫-request处理get
爬虫·python·django
十秒耿直拆包选手8 小时前
Qt:主窗体(QMainwindow)初始化注意事项
c++·qt
用什么都重名10 小时前
MinerU:高效智能PDF文档解析工具完全指南
人工智能·python·pdf·mineru·makedown
霖0010 小时前
C++学习笔记三
运维·开发语言·c++·笔记·学习·fpga开发
mit6.82410 小时前
[shad-PS4] Vulkan渲染器 | 着色器_重新编译器 | SPIR-V 格式
c++·游戏引擎·ps4
倔强青铜三10 小时前
苦练Python第4天:Python变量与数据类型入门
前端·后端·python