Executive Summary
Investment firms - hedge funds, sell-side research groups, traders, and asset managers generally - are increasingly relying on the web as a source of data to improve their investment and trading performance. The motivation is clear: With more than four billion Internet users globally and double-digit zettabytes of data being created annually, the opportunities to gain an investment edge have never been greater.
Opimas estimates that web scraping for investments already accounts for an astounding 5% of all web traffic and that this activity will continue to grow rapidly in coming years.
Table of Contents
|
EXECUTIVE SUMMARY |
|
2 |
|---|---|---|
|
OVERVIEW |
|
3 |
|
|
THE ULTIMATE DATASET |
3 |
|
|
WEB SCRAPING FOR INVESTMENTS |
4 |
|
|
THE WEB-SCRAPING PROCESS |
7 |
|
USE CASES |
|
11 |
|
|
SENTIMENT ANALYSIS |
11 |
|
|
ECONOMIC DATA |
11 |
|
|
AGRICULTURAL DATA |
11 |
|
|
COMPANY REPORTS |
12 |
|
|
MARKET DATA |
12 |
|
|
TRANSPORTATION DATA |
12 |
|
|
ENVIRONMENTAL, SOCIAL AND GOVERNANCE DATA |
12 |
|
|
REAL ESTATE PRICES |
12 |
|
|
CORPORATE ACTIONS |
13 |
|
|
REFERENCE DATA |
13 |
|
|
PRICE COMPARATORS |
13 |
|
|
RETAILERS AND CONSUMER PRODUCTS |
13 |
|
|
TRADING AND INVESTMENT DATA |
14 |
|
SOLUTION PROVIDERS |
|
15 |
|
|
CRUX INFORMATICS |
15 |
|
|
IMPORT.IO |
15 |
|
|
SCRAPINGHUB |
16 |
|
|
SEQUENTUM |
16 |
|
POTENTIAL IMPEDIMENTS |
|
17 |
|
|
REGULATION |
17 |
|
|
JUDICIAL RULINGS |
17 |
|
|
ANTI-WEB-SCRAPING DEFENSES |
18 |
|
LOOKING FORWARD |
|
20 |