python-w3lib¶

Collection of web-related functions for Python (Python 2)

Python module with simple, reusable functions to work with URLs, HTML, forms, and HTTP, that aren’t found in the Python standard library.

This module is used to, for example:

• remove comments, or tags from HTML snippets
• extract base url from HTML snippets
• translate entites on HTML strings
• encoding mulitpart/form-data
• convert raw HTTP headers to dicts and vice-versa
• construct HTTP auth header
• RFC-compliant url joining
• sanitize urls (like browsers do)
• extract arguments from urls

The code of w3lib was originally part of the Scrapy framework but was later stripped out of Scrapy, with the aim of make it more reusable and to provide a useful library of web functions without depending on Scrapy.

This is the Python 2 version of the package.

Related packages: python3-w3lib