Detail scrape is used when you want to scrape a page that has information for only one person, place or thing. Most commonly referred to as a profile page. If you have multiple Detail pages you wish to scrape automatically, that topic is covered in Crawl . In this tutorial, we will cover how to scrape a single page, with the option to manually visit additional pages and collect data as you go.
Detail Scrape Workflow Video
Written Detail Scrape Instructions:
- Navigate to the page you want to scrape, open up Data Miner, click "Start Scraping".
- Select Detail Scrape form the left hand side.
- Click the "Select a Recipe" Tab and choose your recipe. You can choose from Public, Generic, or Private. To test a recipe, simply click on the Recipe and a single scrape will run and data will display on the right hand side of Data Miner.
- If no data is extracted. Continue trying additional Recipes. If no additional Recipes are available scroll down and learn how to write your own Detail Recipe.
- Once you have a recipe that extracts the data you want. Continue to the Scrape Summary tab. This is where you can continue to scrape additional pages using "Scrape again".
- Keeping the Data Miner window open, click back into the browser and manually visit the next detail page you wish to scrape. This means you will have to manually click back in the browser and visit a new page to display new info. Once new data is visible, return to the Data Miner window and click "Scrape again". This will run a single scrape and add it to your data from the previous scrape. Continue this process to scrape additional pages.
- Once you have finished scraping all the desired detail pages, continue to the download tab. From this tab you can download the data directly to your computer as a CSV file, Excel file or you can copy it to your clipboard. Data Miner also has the option to Save it directly in Data Miner. Just give the file a name and click "Save As". This data will be available from the Save Results folder on the left hand side. Once the data is saved or downloaded, you can close Data Miner.
Writing your own Detail Recipe
In this section, we'll quickly cover how to write a Detail Recipe. For more indepth information on Recipe Writing please visit our Recipe Creator section.
- Navigate to a page you want to scrape. We will use this as a template to build the Recipe. Once the recipe is complete, you will be able to use the recipe on any page with the same style for this one website. A new recipe must be build for every new website.
- Once the data is visible, open Data Miner from the toolbar in the top right corner of your browser. Click the little pencil icon. This opens Recipe Creator and starts a new recipe.
- Click into the 2nd tab and choose your page Type. We are currently working on a Detail page. Choose Detail Page.
- Unlike a List Page recipe, you will not have to create rows. You will skip straight to Columns. Columns, is where you target the individual data you wish to scrape.
- To begin selecting the individual data. Click on the Column tab and select “column 1”.
- Give the column a name and pick the extraction type you'd like to perform. By default it is text. What the different extraction types do will be covered in the Recipe Creator section.
- Next, click the "Find" button. This turns your mouse into a magic wand and allows you to select data. To select data, hover your mouse over the actual value you wish to scrape. Once there is a highlighting color covering the data then hit shift on your keyboard.
- Now in Recipe Creator, there will be some selectors suggested. These selectors are pieces of HTML copied from the web page and is what Data Miner uses to locate the data on the page so it can be scarped later. Pick the class or HTML element that highlights the data the best. Do this by click the checkbox in Recipe Creator (Helpful tip - use the "Select Parent" button for more options if the selector outlines too many other items)
- Once the data is highlighted correctly, lock in the selector by clicking "confirm" at the bottom of Recipe Creator and click the "Extracted" button in the corner to check your work.
- Continue creating by clicking "Add new Column".
- Once you have all your columns done, finish by clicking the Save tab at the top. Give the recipe a name, click save and then run to scrape.
Have you created a Detail Recipe, but want to use it on many different pages automatically? This process is covered in our Crawl tutorial.