Browser profiles for web scraping that stays within each site's rules

Start with permission, not tools

Before you write a line of code, answer five questions.

1. Is there an API or an export? Many sites offer an official API, a data feed or a download button. It's faster, more stable and comes with clear terms. Use a browser only when there's no better route.
2. What do the site's terms say? Many sites' terms of service say whether automated access or data collection is allowed. If the terms forbid it, don't scrape that site. Ask the owner for permission or a data feed instead.
3. What does robots.txt say? The Robots Exclusion Protocol, standardized as RFC 9309, lets site owners publish rules that crawlers are "requested to honor". Google's introduction to robots.txt points out that the file can't enforce anything and that obeying it is up to the crawler. So honoring it is your job. Multime doesn't read robots.txt for you.
4. Does the data include people? If you collect information about people, privacy and data protection laws may apply where you or they are. Get legal advice for your case.
5. Whose account is it? Logged-in scraping is for accounts and data you're allowed to access. Downloading your own invoices from a supplier portal is one thing. Collecting other users' data from behind a login is another.

Multime's acceptable use policy forbids collecting personal data unlawfully, scraping in ways that break the law and accessing systems or data without permission.

When a browser profile is the right tool

A plain HTTP client is enough for many jobs. A browser profile helps when:

• The page needs JavaScript. Some dashboards and portals only show data after scripts run, and offer no export.
• Your own data sits behind a login. A supplier portal shows your stock levels only on a web page, or a tool you pay for has reports but no download. A profile keeps the session signed in between runs.
• Content changes by region. You want to see your own product listings, prices and translations the way visitors in each country see them.
• You're testing your own site. Checking pages, forms and language versions from several markets on a schedule.

One profile per job: what goes in it

Each profile is a separate Chromium-based browser with its own cookies, storage, extensions and proxy. Multime supports HTTP, HTTPS, SOCKS5 and SSH proxies, sends DNS through the proxy and runs an IP check that sets time zone, language and location. See the proxy manager and cookie import and export.

Drive profiles from code

Multime's local API runs on your own computer, by default at port 50325, and requires an API key by default. Start a profile, get its DevTools endpoint, and connect Puppeteer, Playwright or Selenium. A short Playwright example that reads a list of your own product pages slowly:

import { chromium } from 'playwright'

const API = 'http://127.0.0.1:50325/api/v1/browser'
const headers = { Authorization: 'Bearer YOUR_API_KEY' }

const res = await fetch(`${API}/start?user_id=PROFILE_ID`, { headers })
const { code, msg, data } = await res.json()
if (code !== 0) throw new Error(msg)

const browser = await chromium.connectOverCDP(data.ws.puppeteer)
const page = await browser.contexts()[0].newPage()

for (const url of urls) {
  await page.goto(url)
  // read only what you need from the page here
  await page.waitForTimeout(5000) // pause between pages
}

await fetch(`${API}/stop?user_id=PROFILE_ID`, { headers })

For Selenium, use the debugger address the API returns with a ChromeDriver that matches the engine version. Current engines are Chrome 152 on Mac and Chrome 150 on Windows. The automation page lists the other endpoints.

Or build it without code

Multime's RPA editor builds flows from steps: open a page, click, type, scroll, wait, read text, take screenshots. Data steps read and write text files and Excel, and integrations write to Google Sheets or call an HTTP API. Logic steps loop over rows of a spreadsheet or over elements on a page. A flow can run once, at an interval, daily or weekly, across many profiles, with a thread limit you set.

A typical flow: read a list of your own product URLs from Excel, open each one in the "Price check – DE" profile, read the displayed price and stock text, wait a few seconds, and write a row to a Google Sheet.

Be a light visitor

A browser profile loads full pages, images and scripts, which costs the site more than an API call. Keep your footprint small:

• Run few profiles at once. Set a low thread limit for small sites.
• Pause between pages. Add waits, and don't fetch the same page twice in one run.
• Run at quiet times. Schedule jobs for off-peak hours in the site's time zone.
• Take no for an answer. If a site rate-limits you, shows a CAPTCHA or blocks you, slow down or stop and ask the owner. Don't switch proxies or profiles to get around it.

Honest limits

• Multime runs on your computer. There's no Multime cloud, so jobs run while your Mac or PC is on. It runs on macOS 13 or later on Apple Silicon and on 64-bit Windows 10 and 11. There's no Linux version, so it doesn't run on a Linux server.
• It isn't a proxy provider or a scraping service. Bring your own proxies, scripts and storage.
• It doesn't make a scraping job allowed. The site's terms, robots.txt and the law decide that.
• Screen size is always your real screen. Plan page layouts and screenshots around it.