This project is a web scraper built using Python, Selenium, and BeautifulSoup to scrape job listings for software engineering roles from Indeed in Sydney, Australia. It extracts in-demand skills and visualizes the data using Seaborn.
- Scrapes multiple pages of job listings from Indeed
- Extracts key skills from job descriptions
- Random wait times to mimic human behavior and avoid bot detection
- Visualizes in-demand skills using Seaborn
To run this project, you'll need to install the dependencies listed in requirements.txt. You can install them with:
pip install -r requirements.txt- Clone the repo
git clone https://github.com/yourusername/job-scraper.git- Navigate to the project directory
cd job-scraper- Create a virtual environment and activate it:
python3 -m venv venv
source venv/bin/activate- Install dependencies
pip install -r requirements.txtTo run the scraper, use the following command
python scraper.py <number_of_pages>Example
python scraper.py 3indeed-scraper-demo.-.Made.with.Clipchamp.mp4
CREDIT TO USER rbuttery I STR8 RIPPED BROS SELENIUM SETUP AND HIS FUNCTION TO CLOSE THE POPUPS ON INDEED :D
This project is licensed under the MIT License. See the LICENSE file for details.