Page Scrape

Once you have determined your page style, you will open the Data Miner window. This is where you run the recipes to perform the scraping.

Data Miner Viewer breakdown

Page scrape is where basic scraping takes place. You must visit a website and run a recipe. In this tutorial, we will cover how to scrape a single page and how to scrape multiple pages. If you wanted Data Miner to visit the website for you, this is covered in the Crawl tutorial.

Page Scrape Overview Video

Written Page Scrape Instructions:

  1. Navigate to the page you want to scrape, open up Data Miner, click "Start Scraping".
    2. Data Miner pop up
  2. Select Page Scrape form the left hand side.
  3. Click the "Select a Recipe" Tab and choose your recipe. You can choose from Public, Generic, or My Recipes. To test a recipe, simply click on the Recipe and a single scrape will run and data will display on the right hand side of Data Miner.
    4. Data Miner Select a Recipe
  4. If no data is extracted. Continue trying additional Recipes. You can refine the recipe list by filtering and searching recipes.
  5. If no additional Recipes are available, you can learn how to write your own by clicking "New Recipe" on the left hand side of Data Miner.
    6. Data Miner Select a Recipe Options
  6. Once you have a recipe that extracts the data you want. Continue to the Scrape tab. This is where you can continue to scrape additional pages using "Scrape Page Again" or "Next Page".
    7. Data Miner Scrape tab

    The "Scrape Page again" button allows you to manually visit a new page of the same style and scrape new data into the same file. "Next Page Pagination" is when you want Data Miner to automatically click a Next button and scrape data. (Only available for List Recipes.)

    Data Miner Scrape tab
  7. Once you have finished scraping all the desired detail pages, continue to the download tab. From this tab you can download the data directly to your computer as a CSV file, Excel file or you can copy it to your clipboard. Data Miner also has the option to Save it directly in Data Miner. Just give the file a name and click "Save As". This data will be available from the Save Results folder on the left hand side. Once the data is saved or downloaded, you can close Data Miner.
    8. Data Miner Download Tab

Try on your own!

Using our Practice Sandbox try running through the above steps on your own!

Additional Documentation

Do you have to click into a Detail page to access the data? This is covered in the Crawl section.

Do you not see any Public Recipes? You can learn to make recipes yourself from our Recipe Creator section.