Mission Statement
- Crawling-for-dummies develops open-source software for web crawling service.
- Crawling-for-dummies is a chrome-addons that allows user-friendly approach to web crawling.
- It is design to easy-to-use software that can be used with little knowledge.
What is web crawling?
Web crawling is the process of indexing information on a website using automated scripts or programs. These programs are called by various names, including Web crawlers, Spider, Spider-bot, and Crawlers.
Why web crawling is IMPORTANT?
In 2013, IBM announced that approximately 90% of the world’s data was generated in the last two years and that the rate of data generation doubled every two years. However, most data is unstructured and Web crawlers index vast amounts of unstructured data to help search engines find the information they want to find.
Indexed data also plays an important role in various fields - data science projects.
In our project…
Crawlers that currently exist are hardly accessible and require professional coding, making it difficult for non-major to use. In addition, the existing crawler program has the disadvantage that it is difficult to grasp at once because the UI is not intuitive. Therefore, we want to improve UI and UX so that users who have nothing to do with coding can easily use crawling programs.
Target Development Language
- Technology stack: Node.js, Manifest V3
- Supports Windows 10 / Mac OS Environment
- dependency manager: Yarn, NPM
- development tool: VSCode, Chrome
- version control system: Git, Docker
- project management tool: discord, notion
- project document: read the docs
- Enough disk space to hold the crawling data