Skip to content
Webcrawler API docs

Job

Job - is a task that you can run on the Webcrawler API. It has an asynchronous nature. It means you will get a notification when it is done (read more about async request).

Job request parameters

  • url - (required) the seed URL where the crawler starts. Can be any valid URL.
  • scrape_type - (default: html) the type of scraping you want to perform. Can be html, cleaned.
  • items_limit - (default: 20) crawler will stops when it reaches this limit of pages for this job.
  • webhook_url - (optional) the URL where the server will send a POST request once the task is completed (read more about webhooks and async requests).
  • crawl_delay - (default: 2000) delay between requests in milliseconds. To respect the website and avoid being blocked we recommend to leave it default.
  • max_retries - (default: 2) the number of retries if page request fails.
  • whitelist_regexp - (optional) a regular expression to whitelist URLs. Only URLs that match the pattern will be crawled.
  • blacklist_regexp - (optional) a regular expression to blacklist URLs. URLs that match the pattern will be skipped.
  • scrape_type - (default: html) the type of scraping you want to perform. Can be html, cleaned.
  • allow_subdomains - (default: false) if true the crawler will also crawl subdomains (for example, blog.example.com if the seed URL is example.com).

Example:

{
    "url": "https://stripe.com/",
    "webhook_url": "https://yourserver.com/webhook",
    "items_limit": 10,
    "crawl_delay": 2000,
    "max_retries": 1,
    "scrape_type": "clean",
    "allow_subdomains": false
}

Job response

  • id - the unique identifier of the job.

  • url - the seed URL where the crawler started.

  • status - the status of the job. Can be new, in_progress, done, error.

  • scrape_type - the type of scraping you want to perform.

  • extract_rules - an object with rules to extract data from the page.

  • whitelist_regexp - a regular expression to whitelist URLs.

  • blacklist_regexp - a regular expression to blacklist URLs.

  • allow_subdomains - if the crawler will also crawl subdomains.

  • items_limit - the limit of pages for this job.

  • crawl_delay_ms - delay between requests in milliseconds.

  • max_retries - the number of retries if page request fails.

  • created_at - the date when the job was created.

  • finished_at - the date when the job was finished.

  • webhook_url - the URL where the server will send a POST request once the task is completed.

  • webhook_status - the status of the webhook request.

  • webhook_error - the error message if the webhook request failed.

  • job_items - an array of items that were extracted from the pages.

    Job Item:

    • id - the unique identifier of the item.
    • status - the status of the item. Can be new, in_progress, done, error.
    • job_id - the job identifier.
    • original_url - the URL of the page.
    • page_status_code - the status code of the page request.
    • raw_content_url - the URL to the raw content of the page.
    • cleaned_content_url - the URL to the cleaned content of the page (if scrape_type is cleaned).
    • title - the title of the page.
    • created_at - the date when the item was created.
    • cost - the cost of the item in $.

Example:

{
  "id": "23b81e21-c672-4402-a886-303f18de9555",
  "url": "https://stripe.com/",
  "scrape_type": "clened",
  "extract_rules": "",
  "whitelist_regexp": "",
  "blacklist_regexp": "",
  "allow_subdomains": false,
  "items_limit": 10,
  "created_at": "2024-06-17T12:22:08.034Z",
  "crawl_delay_ms": 0,
  "finished_at": "2024-06-17T12:23:01.53Z",
  "webhook_url": "https://yourserver.com/webhook",
  "webhook_status": 0,
  "webhook_error": "",
  "status": "done",
  "job_items": [
    {
      "id": 578720,
      "job_id": "23b81e21-c672-4402-a886-303f18de9555",
      "original_url": "https://stripe.com/docs/no-code/tap-to-pay",
      "page_status_code": 200,
      "raw_content_url": "https://data.webcrawlerapi.com/raw/clwgv3ywz000hsy99lwbk7q18/23b81e21-c672-4402-a886-303f18de9555/https___stripe_com_docs_no_code_tap_to_pay",
      "cleaned_content_url": "https://data.webcrawlerapi.com/raw/clwgv3ywz000hsy99lwbk7q18/23b81e21-c672-4402-a886-303f18de9555/https___stripe_com_docs_no_code_tap_to_pay",
      "status": "done",
      "title": "Tap to Pay on the Dashboard mobile app | Stripe Documentation",
      "created_at": "2024-06-17T12:22:19.511Z",
      "updated_at": "2024-06-17T12:22:33.334Z",
      "retries": 0,
      "cost": 0.002
    }
    ]
}