Building a job-search pipeline with Playwright, SQLite, and Python

Looking for a developer job in Warsaw means dealing with Pracuj.pl, the dominant Polish job board. Their search filters are coarse, their results page is paginated, and every offer has structured data hidden inside the HTML that the UI only partly exposes. I wanted to ask questions like “backend roles in Warsaw, hybrid, paying over 18k PLN, with .NET in the optional tech list” — and get answers in seconds, not by clicking through 40 listings. So I built a scraper.

The result is three files: a Node script that drives Playwright through 200 offer pages, a Python script that normalizes the JSON output into SQLite, and a query script for the day-to-day filtering. ~600 lines total. I want to walk through the four decisions that mattered.

Why Playwright instead of fetch

The first thing I tried was the obvious thing: fetch(url), parse the HTML, extract what I needed. That hit two walls in the first ten requests.

Wall one: Cloudflare. Pracuj sits behind Cloudflare's bot protection, which serves a JavaScript challenge before the real page renders. A plain HTTP client never sees the offer — it sees the challenge HTML. You can solve this with custom headers and rotating user agents up to a point, then you can't.

Wall two: client-side rendering. Even past Cloudflare, the offer details — required technologies, salary breakdown, position level — are populated by client-side React after the initial HTML loads. cheerio over the raw response gets you a skeleton with empty divs.

Playwright solves both at once. It runs a real Chromium, executes the JavaScript, waits for Cloudflare's challenge to clear, and exposes the fully-rendered DOM. The cost is real — each page takes ~12 seconds end to end versus ~200ms for a fetch — but the alternative isn't a faster scrape, it's no scrape.

const { chromium } = require('playwright');

const PAGE_WAIT_MS = 10000;     // wait for Cloudflare challenge
const BETWEEN_PAGES_MS = 2500;  // polite delay between pages

async function scrape(urls) {
  const browser = await chromium.launch();
  const ctx = await browser.newContext();
  for (const url of urls) {
    const page = await ctx.newPage();
    await page.goto(url, { waitUntil: 'networkidle' });
    await page.waitForTimeout(PAGE_WAIT_MS);
    const data = await extractOfferData(page);
    await page.close();
    await page.waitForTimeout(BETWEEN_PAGES_MS);
  }
}

The 2.5-second delay between pages is not politeness theater. It's the difference between completing a 200-offer run and getting throttled at offer 80.

The DOM strategy: anchor on data-scroll-id

Class names on big React sites are useless for scraping. Pracuj's are emitted by a CSS-in-JS pipeline — t1u8helo, c1bf6vbi — and they change between deploys. Anchoring on those class names buys you a scraper that breaks every two weeks.

What survives is the semantic attributes. Pracuj uses data-scroll-id on every section heading because the page's anchor-link navigation needs them. Those IDs are descriptive (requirements-expected-1, position-levels, responsibilities-1) and they don't churn. Twelve sections, twelve stable anchors.

const scrollSections = [
  'work-schedules', 'position-levels', 'work-modes',
  'about-project-1', 'responsibilities-1',
  'requirements-expected-1', 'requirements-optional-1',
  'offered-1', 'benefits-1',
  'development-practices-1',
  'about-us-description-1',
  'attribute-secondary-it-specializations',
];

for (const sectionId of scrollSections) {
  const el = document.querySelector(`[data-scroll-id="${sectionId}"]`);
  if (el) {
    const items = el.querySelectorAll('li, [data-test*="item"]');
    sections[sectionId] = items.length
      ? [...items].map(i => i.textContent.trim()).filter(Boolean)
      : el.textContent.trim();
  }
}

There's also the JSON-LD block at the top of every offer — the structured data Pracuj emits for Google's job-posting rich results. That's the canonical source for company, salary, employment type, and dates. The DOM-scraped sections fill in everything that doesn't fit the schema.org JobPosting schema.

The DOM is unreliable; the schema.org JSON-LD is contractual. Read both, prefer the contract.

Why SQLite, not Postgres

The instinct in 2026 is to reach for a managed Postgres. For a personal scraper, that's the wrong instinct. SQLite gets four things right that matter at this scale:

1. The database is a file. job_offers.db lives next to the scraper. I can .gitignore it. I can rm it to reset. I can cp it to a USB drive. There is no service to start, no port to expose, no credentials to rotate.

2. Tools that already speak it. sqlite3 job_offers.db from the terminal is a complete query environment. DBeaver opens it without a connection string. Python's stdlib has sqlite3. No driver to install.

3. WAL mode handles the only concurrency I need. The scraper writes; the query script reads. WAL (PRAGMA journal_mode=WAL) lets the reader see consistent snapshots while the writer is appending, which is the entire concurrency story for this app.

4. The schema is normalized properly anyway. SQLite isn't a downgrade from Postgres for tabular data. The schema below has foreign keys, cascading deletes, and a unique constraint preventing duplicate technologies per offer.

CREATE TABLE offers (
  id              INTEGER PRIMARY KEY AUTOINCREMENT,
  pracuj_id       TEXT UNIQUE,
  url             TEXT NOT NULL,
  title           TEXT,
  company         TEXT,
  city            TEXT,
  salary_min      REAL,
  salary_max      REAL,
  salary_currency TEXT,
  position_level  TEXT,
  work_mode       TEXT,
  date_posted     TEXT,
  raw_json_ld     TEXT
  -- ...20 more fields
);

CREATE TABLE offer_technologies (
  offer_id   INTEGER REFERENCES offers(id) ON DELETE CASCADE,
  technology TEXT NOT NULL,
  required   BOOLEAN DEFAULT 1,
  UNIQUE(offer_id, technology)
);

Why Node and Python in the same folder

The scraper is Node. The database build script is Python. The query CLI is Python. Mixing runtimes inside one project is usually a smell. Here it's a feature, because each runtime is doing what its ecosystem is best at.

Playwright's Node API is the most-used and best-documented surface. Selector debugging, codegen, the trace viewer — all of them exist for the Node API and are hand-me-downs for the Python port. For a scraper that will break when the site changes, you want the best debugging tools, not the most-cohesive language choice.

Python's stdlib is built for this kind of data work. sqlite3, re for text cleanup (Pracuj's job descriptions are full of emoji bullets I want to strip), json for the JSON-LD payloads, argparse for the query CLI. Node can do all of these but everything is a library install. Python ships with a working version on day one.

BULLET_RE = re.compile(
    r"^[\s]*[•✔️✅❌⭐🔹🔸💻🛠️📌▪▸►‣⁃–—\-\*]\s*",
    re.MULTILINE,
)

def clean_text(val):
    if not val:
        return val
    if isinstance(val, list):
        return ", ".join(clean_text(v) for v in val if v)
    return BULLET_RE.sub("", str(val))

The interface between the two is a single scraped_offers.json file. The Node side's only job is “produce that file”; the Python side's only job is “read that file.” There's no shared state, no shared types, no foreign function call. The seam is just JSON and the filesystem — the most portable interface in computing.

What the pipeline actually produces

After a scrape run I get a SQLite database with a few hundred rows. The query CLI is built around the questions I actually ask:

— “Which Warsaw backend roles list .NET as a required technology and pay over 18k PLN?”

— “Which companies have posted three or more roles in the last 30 days?”

— “Show me hybrid roles tagged 'mid' that mention React in the optional list.”

These are SQL joins with WHERE clauses, indexed on position_level, work_mode, salary_min, and the offer_technologies table. They run in milliseconds. The cover-letter generator (different post) reads from the same database to tailor each application.

The pipeline is small enough to be in one folder, weird enough that nobody else has built it, and useful enough that I run it weekly. That's the right shape for a tool you make for yourself: do one specific thing nobody else cares about, do it offline, do it in whatever language is best at the part you're doing.