The job search process in the software industry presents a classic data engineering challenge: ingesting, normalizing, filtering, and processing information from highly heterogeneous and unstructured sources. This article details the design, implementation logic, and complete event-driven software architecture built to support an efficient, accurate, and ethical job search.
Using n8n as a self-hosted workflow orchestration engine, Supabase (PostgreSQL) as the relational persistence layer, advanced responsible scraping logic, Artificial Intelligence agents (LLMs via the OpenAI API with structured outputs) for technical-fit evaluation, and a Telegram Bot as an interactive notification interface, an industrial-grade data pipeline was built. Finally, the system integrates with a personal website to consume normalized data and render resume variants as optimized PDF documents.
Throughout this document, we take an in-depth look at the database scripts, granular automation-node configuration, infrastructure trade-offs, and strategies for maintaining strict human control (human-in-the-loop) throughout the entire process lifecycle.
2. The problem: job searching is also an information problem
For a software engineer, an active job search often becomes a high-friction operational task. The traditional workflow requires browsing multiple platforms every day (such as Hireline, GetOnBoard, or dedicated job portals), manually evaluating ambiguous job descriptions, filtering duplicate postings, and adapting the resume's focus to highlight the technologies relevant to each employer.
From a systems perspective, this process suffers from three critical inefficiencies:
- Fragmentation and lack of structure: Each job portal renders information using different schemas. Job descriptions are free-form text, which makes immediate parsing with traditional software tools difficult.
- Latency and asynchrony: The job market is dynamic. Important opportunities can appear and close within hours, requiring continuous monitoring that consumes cognitive bandwidth.
- Document boilerplate: Manually changing the hierarchy of a PDF or JSON file to emphasize experience with TypeScript, distributed architectures, or frontend development according to the target role is repetitive and prone to consistency errors.
If we treat job searching as a data-ingestion pipeline, we can design an automated solution that extracts, unifies, and pre-evaluates these signals, allowing the engineer to focus exclusively on high-value human interaction: interviews and in-depth technical validation.
3. What I wanted to automate — and what I did not
Before writing the first line of code or configuring the first orchestration node, it was essential to define the system's ethical and operational boundaries. The primary goal of AI automation in this project is not mass job applications or automated application submission (spamming), a practice that degrades recruiting processes and damages the candidate's reputation.
Automated processes:
- Chronological discovery of job openings across multiple web platforms through scheduled tasks (cron jobs).
- Extraction of the raw job-posting text and its subsequent transformation into a strict JSON data schema.
- Immediate rejection of duplicate postings or opportunities that violate explicit constraints (for example, 100% on-site roles in geographically impractical locations).
- Calculation of a technical-fit indicator (match score) based on overlap between the technology stack and required years of experience versus the engineer's actual profile.
- Preparation of personalized document artifacts (structured data ready to be rendered as a PDF resume).
Strictly manual processes (Human-in-the-loop):
- The final decision to apply for a job.
- Review and fine-tuning of the resume focus suggested by the AI.
- Writing personalized introductory messages to recruiters.
- Interaction throughout the interview stages.
Automation should be conceived as an augmented-intelligence tool: repetitive data-processing tasks are delegated to machines, while analytical, ethical, and strategic judgment remains with the human.
4. Overall system architecture
The solution was designed around a decoupled topology in which state and transactions live in the database, while business logic and external integrations are managed by the workflow orchestrator.
The following diagram shows the logical flow of data through the different components of the system:
Job Boards / Sources (Hireline, GetOnBoard, etc.)
│
▼ [Cron Job / HTTP Request]
Scraping / API Ingestion
│
▼ [Raw HTML / Semi-structured JSON]
n8n Workflow (Orchestrator)
│
▼ [Normalization & JSON Schema Validation]
Supabase / PostgreSQL (Data Layer: Unique Constraints)
│
▼ [Unprocessed Queue Trigger]
AI Match Scoring (OpenAI API / Structured Outputs)
│
▼ [Payload Update & Score Storage]
Telegram Notification (Inline Webhook Buttons)
│
▼ [Human Review: Click to Approve/Reject]
Dynamic PDF Resume Generation (Personal Website Web Engine Render)The data lifecycle begins with the periodic activation of a time-based trigger in n8n. It retrieves recent job postings, processes them to verify integrity, and inserts them into Supabase. Once persisted, the workflow invokes the natural-language-processing capabilities of a large language model (LLM) to enrich each record with analytical metadata. Finally, the messaging interface acts as the decision bottleneck where the user approves or rejects the opportunity.
5. Data sources and responsible scraping
Job-posting ingestion was implemented by combining public/semi-public APIs with web-scraping techniques on structured job portals. To ensure system sustainability and respect for third-party infrastructure, strict responsible scraping guidelines were established:
- Respect for exclusion directives: The rules defined in each target domain's /robots.txt file are checked in advance.
- Concurrency control and rate limiting: Instead of launching massive parallel requests that could resemble a denial-of-service (DoS) attack, the orchestrator uses sequential loops with randomized Wait Nodes of between 2 and 5 seconds between HTTP requests.
- Agent identification: Requests use clear and descriptive User-Agent headers, avoiding malicious browser impersonation.
- Load minimization: Hashes of already processed URLs are stored locally to avoid downloading pages that have already been inspected in previous runs.
When a web portal does not expose a readable data API (JSON), the HTML DOM tree is extracted using robust CSS selectors, prioritizing plain-text containers (such as <article> sections or description blocks) to reduce the fragility caused by changes to a platform's user-interface design.
The following JavaScript snippet is used in an n8n processing node to clean and parse raw HTML returned by an HTTP request:
// n8n node: Code Node (JavaScript)
// Input: Raw HTML obtained from the HTTP Request node
const items = $input.all();
const output = [];
for (const item of items) {
const htmlContent = item.json.body;
// Regular expressions used to extract semantic blocks when a full DOM parser is unavailable
// In production n8n environments, the internal library or native selectors from the 'HTML' node can be used
// Basic cleanup of scripts, styles, and irrelevant tags to reduce token usage
let cleanText = htmlContent
.replace(/<script[^>]*>[\s\S]*?<\/script>/gi, '')
.replace(/<style[^>]*>[\s\S]*?<\/style>/gi, '')
.replace(/<[^>]+>/g, ' ')
.replace(/\s+/g, ' ')
.trim();
output.push({
json: {
rawText: cleanText,
extractedAt: new Date().toISOString()
}
});
}
return output;6. Job-posting normalization
Job descriptions vary drastically in format. One recruiter may list the technology stack as bullet points, while another may describe it narratively inside a long paragraph. The goal of the normalization stage is to transform this textual chaos into a predictable data entity that conforms to a strict JSON schema.
A normalized payload ready to be inserted into the operational database has the following structure:
{
"title": "Full-Stack Engineer",
"company": "Example Company",
"location": "Remote",
"stack": ["React", "Node.js", "PostgreSQL", "TypeScript"],
"seniority": "Mid/Senior",
"sourceUrl": "https://example.com/job/12345",
"matchScore": 0.82
}To achieve this level of structure from raw HTML, the system uses optimized OpenAI API calls while constraining the model's response to specific data schemas. To make this step robust, schema validation is implemented with JSON Schema in the OpenAI API call. The following schema defines the data types required by the backend:
{
"type": "object",
"properties": {
"title": { "type": "string" },
"company": { "type": "string" },
"location": { "type": "string" },
"stack": {
"type": "array",
"items": { "type": "string" }
},
"seniority": { "type": "string" },
"sourceUrl": { "type": "string" }
},
"required": ["title", "company", "location", "stack", "seniority", "sourceUrl"],
"additionalProperties": false
}This ensures that subsequent stages of the backend architecture consume uniform primitive data types, abstracting away variability in the source data.
7. Supabase as the database and operational backend
System persistence was entrusted to Supabase, leveraging the robustness of PostgreSQL for state management and relational data integrity. The database is not used as a simple read/write store; it acts as the state machine governing the entire automation pipeline.
The main table was defined using the following SQL migration script:
create table job_opportunities (
id uuid primary key default gen_random_uuid(),
title text not null,
company text not null,
location text,
source_url text unique not null,
description text,
seniority text,
stack text[] default '{}',
match_score numeric check (match_score >= 0 and match_score <= 1),
status text default 'new' check (status in ('new', 'under_review', 'cv_generated', 'applied', 'rejected', 'interviewing', 'archived')),
created_at timestamptz default now()
);
-- Indexes for optimizing operational and filtering queries
create index idx_job_opportunities_status on job_opportunities(status);
create index idx_job_opportunities_match_score on job_opportunities(match_score);Key PostgreSQL design characteristics:
- Idempotency through unique constraints: The source_url column has a UNIQUE constraint. When the scraper attempts to reinsert an existing job posting during later daily runs, the database rejects the operation (ON CONFLICT DO NOTHING). This prevents duplicate records without requiring expensive application-level verification logic.
- Integrity validation: The status field is restricted by a Check Constraint that ensures application states strictly follow the defined lifecycle, preventing data corruption caused by invalid states.
- Strategic indexing: B-Tree indexes were created on the status and match_score columns because the system constantly reads records filtered by the new state and sorts them by score relevance.
To extend Supabase's operational capabilities and automate post-insert actions, a Database Trigger can notify external services or execute immediate logic whenever an opportunity with a high potential score is inserted. The following script shows how to create an internal function that audits these inserts:
create or replace function log_new_high_match_opportunity()
returns trigger as $$
begin
if new.match_score >= 0.80 then
raise log 'High technical-fit alert detected: %, Company: %', new.title, new.company;
end if;
return new;
end;
$$ language plpgsql;
create trigger trigger_high_match_opportunity
after insert or update on job_opportunities
for each row
execute function log_new_high_match_opportunity();8. n8n as the workflow orchestrator
n8n was selected as the orchestration core because the system needs to coordinate multiple heterogeneous services (scrapers, databases, LLMs, and Telegram APIs) while maintaining visibility into each execution and enabling fast visual debugging. Unlike solutions implemented entirely in code or simplified platforms such as Zapier, n8n provides granular control over both data and control flow in self-hosted environments (Ubuntu Server via Docker).
Analytical breakdown of workflow nodes:
- Cron Trigger (Scheduling): Runs the workflow automatically every day at 08:00 AM and 06:00 PM (0 8,18 * * *).
- HTTP Request (Ingestion): Sends authenticated or structured GET requests to job-portal endpoints while injecting dynamic User-Agent headers.
- Item Lists (Split In Batches): Takes the multidimensional array of discovered job postings and splits it into individual elements (batches of size 1) to enable controlled sequential processing.
- Supabase Node (Initial Insert): Attempts to insert the basic data (title, company, source_url) into the job_opportunities table. It uses safe insertion semantics. If the URL already exists, the database aborts the insert for that specific record without stopping the overall workflow because the node is configured with Continue On Fail = True.
- If Node (Duplicate Check): Determines whether the previous node inserted a new row or returned a uniqueness-constraint error. If it is a duplicate, processing for that item ends immediately (Graceful Exit).
- OpenAI Node (Analytical Evaluation): Sends the plain-text job description to the gpt-4o model configured with strict response parameters.
- Supabase Node (Update): Updates the database record with the match_score, the cleaned list of technologies in the stack, and the quantitative fit breakdown.
- Telegram Node (Notification): If the calculated score exceeds the threshold, it sends a message with interactive buttons directly to the software engineer's chat.
- Wait Node (Courtesy Delay): Pauses the workflow for 3 seconds before continuing with the next job posting in the data loop, reducing network load.
9. AI-based match evaluation
Qualitatively evaluating a job posting traditionally requires the engineer to read dense paragraphs and determine whether their backend or frontend skills align with the role's expectations. In this system, that task is delegated to an AI agent configured to behave deterministically.
The n8n workflow retrieves the engineer's professional profile (stored as a static technical reference document in structured format) and the newly ingested job description, then sends both data blocks to a language model through the OpenAI API.
The evaluation system's internal typing logic is defined in TypeScript using the following structure:
type JobMatchInput = {
jobTitle: string;
description: string;
requiredSkills: string[];
profileSkills: string[];
}
type JobMatchResult = {
score: number; // Floating-point value between 0.0 and 1.0
reasons: string[];
missingSkills: string[];
suggestedResumeFocus: string[];
}Structural prompt injected into the OpenAI node:
You are a technical evaluator specializing in software architecture and engineering talent assessment.
Your task is to objectively and analytically evaluate the match between the Candidate Profile and the attached Job Description.
Candidate Profile:
- Core technologies: TypeScript, JavaScript, Node.js, React, Supabase, PostgreSQL, Docker, n8n.
- Experience: Full-Stack development focused on event-driven architecture, database optimization, and SPA frontend development.
Instructions:
1. Strictly compare the technical requirements against the candidate profile.
2. If the job requires technologies for which the candidate has no verifiable experience, add them to 'missingSkills'.
3. Generate a numeric 'score' based purely on the overlap between the technology stack and the required seniority level.
4. Generate honest suggestions in 'suggestedResumeFocus' describing which real areas of the candidate's profile should receive visual priority if they decide to apply.
Process the following job posting:
Title: {{ $json.title }}
Description: {{ $json.description }}The model does not "invent" experience. Its cognitive role is limited to intersecting skill sets (profileSkills vs requiredSkills), identifying critical gaps, and generating a recommendation vector (suggestedResumeFocus) that determines which subsets of the candidate's real experience should be highlighted during the document-generation phase.
10. Telegram as the notification and review interface
Using complex web applications to manage internal tools often adds unnecessary maintenance overhead. Instead, this system uses a Telegram Bot as a minimal, asynchronous, mobile-friendly user interface.
When a record is processed and receives a match_score above the configured minimum threshold (for example, greater than 0.70), n8n builds a messaging payload and sends an HTTP request to the Telegram API. The notification received on the user's mobile device is carefully structured to support fast decisions:
New opportunity detected
Role: Full-Stack Engineer
Company: Example Company
Match: 82%
Stack: React, Node.js, PostgreSQL
Reasons:
- Strong React/Next.js alignment
- Backend and PostgreSQL experience required
- Remote role
Actions:
[Review] [Generate CV] [Ignore]The bottom button row uses Telegram Inline Keyboards. Each button is associated with a return-data string (callback_data) containing the unique Supabase ID of the job posting (for example, cv_gen:job-posting-uuid).
When a button is pressed, Telegram sends a webhook directly to an endpoint exposed by n8n or a Supabase Edge Function, immediately triggering the next stage of the workflow without requiring the user to open a browser or type commands in a terminal.
11. Dynamic PDF resume generation
Once the user presses the [Generate CV] button in Telegram, the system begins compiling an adapted resume. The ethical principle behind this design must be reiterated: adapting is not falsifying. The system never invents jobs, responsibilities, or technologies the candidate does not actually know. The process consists exclusively of changing the information architecture of the original document.
Technical generation mechanism:
- The system retrieves the user's master professional-profile JSON, which contains their complete work history, software projects, and verified technical skills.
- The suggestedResumeFocus array previously generated by the LLM is processed. If the role focuses on backend development and relational databases, the generator reorders resume sections to place Supabase, PostgreSQL, and Node.js projects near the top, while reducing detail or compacting projects focused exclusively on frontend layout or visual-design libraries without removing them.
12. Integration with the personal website and rendering
To complete the workflow and ensure a polished, professional visual result for recruiters, the final stage of the pipeline integrates directly with the engineer's personal website. Instead of relying on opaque third-party generators, the web portfolio acts as the official rendering engine, consuming the normalized payloads stored in Supabase.
A secure endpoint was built in the personal website backend (using a Supabase Edge Function or a Node.js/TypeScript API route) to receive compilation requests, orchestrate the on-screen rendering of the resume structure using standard React/Next.js components or clean semantic HTML, and export the result as a binary PDF file through Puppeteer in headless mode.
The following is the complete source code of the Edge Function responsible for retrieving the structured profile data and transforming it dynamically:
// Supabase Edge Function: generate-pdf-cv/index.ts
import { serve } from "https://deno.land/std@0.168.0/http/server.ts"
import { createClient } from "https://esm.sh/@supabase/supabase-client@2"
const CORS_HEADERS = {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Headers': 'authorization, x-client-info, apikey, content-type',
}
serve(async (req) => {
// Handle preflight requests for CORS policies
if (req.method === 'OPTIONS') {
return new Response('ok', { headers: CORS_HEADERS })
}
try {
const supabaseClient = createClient(
Deno.env.get('SUPABASE_URL') ?? '',
Deno.env.get('SUPABASE_SERVICE_ROLE_KEY') ?? ''
)
// Extract the job opportunity ID sent from the n8n/Telegram webhook
const { opportunityId } = await req.json()
// 1. Retrieve analytical information for the job opportunity
const { data: job, error: jobError } = await supabaseClient
.from('job_opportunities')
.select('*')
.eq('id', opportunityId)
.single()
if (jobError || !job) throw new Error('Job opportunity not found')
// 2. Retrieve the candidate's static master profile
// Assume it lives in a 'candidate_profiles' table
const { data: profile, error: profileError } = await supabaseClient
.from('candidate_profiles')
.select('*')
.limit(1)
.single()
if (profileError || !profile) throw new Error('Master profile not found')
// 3. Reordering logic based on AI suggestions (suggested_resume_focus)
let dynamicExperience = [...profile.experience]
const coreFocus = job.stack || []
// Sorting algorithm based on technical relevance
dynamicExperience.sort((a, b) => {
const aMatch = a.technologies.some((tech: string) => coreFocus.includes(tech)) ? 1 : 0
const bMatch = b.technologies.some((tech: string) => coreFocus.includes(tech)) ? 1 : 0
return bMatch - aMatch // Prioritize the role that matches the target stack
})
// Final structured payload ready to be consumed by the portfolio design engine
const cvPayload = {
name: profile.full_name,
title: job.title, // Adapt the heading to the specific target role
skills: profile.skills,
experience: dynamicExperience,
metadata: {
generatedFor: job.company,
sourceOpportunity: job.id
}
}
// In a production environment, this payload is sent to an internal Chromium instance
// to compile the HTML with CSS Print Media into a PDF. Here we return a successful build response.
return new Response(
JSON.stringify({ success: true, message: "Resume payload optimized successfully", data: cvPayload }),
{ headers: { ...CORS_HEADERS, "Content-Type": "application/json" }, status: 200 }
)
} catch (error: any) {
return new Response(
JSON.stringify({ error: error.message }),
{ headers: { ...CORS_HEADERS, "Content-Type": "application/json" }, status: 400 }
)
}
})The final visual rendering in the web portfolio uses strict print-media CSS rules (@media print), configuring exact dimensions in centimeters (A4 or Letter) and removing unnecessary interactive layout elements such as global navigation bars or website footers. This ensures that the resulting PDF maintains a professional, executive-level presentation suitable for automated resume-screening systems (ATS).
13. Human-in-the-loop: review before applying
The automation system is a high-speed data assistant, but it lacks contextual intuition, deep understanding of organizational culture, and fine-grained verification of language nuances. For that reason, the human-review stage is the project's core quality-control mechanism.
The manual verification process before formally submitting an application focuses on auditing three areas:
- LLM hallucination validation: Although structured outputs reduce the margin for error, the user verifies that the AI has not misinterpreted a negation in the job description (for example, a posting stating "We are not looking for PHP developers" while the model incorrectly extracts PHP as a mandatory requirement).
- Tone refinement: Ensuring that the key points highlighted in the resume maintain natural, fluent prose aligned with the engineer's professional identity.
- Company-fit confirmation: A quick review of the hiring company's website to validate qualitative factors that scraping tools cannot capture reliably.
If the quality-control review is successful, the user proceeds with the application manually or interacts with the job portal using the optimized PDF file generated by the architecture within seconds.
14. Statuses, tracking, and traceability
A critical secondary benefit of centralizing this process in a robust relational database such as PostgreSQL is the ability to audit the complete traceability of the job-application funnel. The status field in the job_opportunities table allows the system to reliably map the state of each interaction.
[new] ──► [under_review] ──► [cv_generated] ──► [applied] ──► [interviewing] ──► [archived]
│ │ │
▼ (Manual/AI rejection) ▼ (Rejected) ▼ (No offer)
[rejected] [rejected] [rejected]When the user performs actions through Telegram webhooks, n8n updates the corresponding row in the database:
- If [Ignore] is pressed, the status changes to rejected, creating historical data that can later be used to analytically exclude similar roles from active queries.
- When the resume is generated, the status changes to cv_generated.
- When the user confirms that they have completed the application form on the external platform, the system marks the record as applied.
This analytical persistence enables the creation of simple monitoring dashboards that calculate pipeline performance metrics: response rate by technology-stack category, funnel-processing speed, and geographic distribution of job postings with the highest real technical fit.
15. Security, privacy, and boundaries
When building automation systems that handle sensitive personal data and interact with infrastructure credentials, security cannot be treated as a secondary concern.
Implemented mitigation measures:
- Network isolation: The n8n instance and operational database run in a controlled private environment that is externally accessible only through a secure overlay network (Tailscale). Webhooks exposed to the public internet to receive Telegram requests use obfuscated endpoints and strict validation of authorization tokens in HTTP headers.
- Secret management: No API key (OpenAI Secret Key, Supabase Service Role Key, or Telegram Bot Token) is written directly into scripts or nodes. Secrets are injected strictly as secure environment variables in the Docker container or managed through n8n's native encrypted credential store.
- PII (Personally Identifiable Information) protection: Scraping stages clean and remove unnecessary personally identifiable information before sending payloads to third-party servers for AI analysis. Final PDF files stored in Supabase buckets are protected with row-level security policies (RLS - Row Level Security), preventing unauthorized public access to application documents.
16. Common mistakes when automating this process
Developing this system made it possible to document common design failures in workflow automation. Identifying them is essential for any automation engineer aiming to build stable solutions:
- Blind trust in DOM consistency: Basing scraping exclusively on hyper-specific CSS selectors (for example, div.container > div.row > span.text-sm) causes the pipeline to break whenever the web platform makes even a minor UI update. Solution: Consume internal job-portal APIs by inspecting network requests whenever feasible, or use flexible regular expressions combined with broad semantic-container selectors.
- Lack of LLM input-volume control: Sending extremely long job descriptions to the language model—including company legal text, diversity policies, or site terms of use—dramatically increases unnecessary token consumption. Solution: Build a JavaScript/TypeScript preprocessing node that removes HTML tags, scripts, styles, and repetitive text before invoking the OpenAI API.
- Lack of idempotency in n8n: If a workflow fails halfway through execution (for example, during the Telegram bot call) and the node retries the entire execution, the system risks creating duplicate database inserts or consuming AI credits again. Solution: Ensure that the database insert using a unique key occurs early in the workflow; if insertion fails because the record already exists, the workflow should terminate cleanly and immediately (graceful exit).
17. Lessons learned
Designing and operating this automation architecture provides valuable software-engineering lessons that extend beyond the job-search use case:
- Visual workflows still require code discipline: Tools such as n8n simplify system integration, but without modular structure, visual workflows become unmaintainable ("visual spaghetti code"). Processes should be broken into independent, specialized sub-workflows.
- The importance of data decoupling: Designing the system so that n8n stores no internal state and instead relies entirely on transactions persisted in PostgreSQL (Supabase) ensures that if the automation server restarts or crashes, operational consistency is not lost; the pipeline can resume by inspecting the database state.
- AI as a software component, not an oracle: Language models are excellent tools for format transformation and textual feature extraction, but delegating complete business-logic decisions to them without structural constraints produces erratic systems. Combining hard deterministic rules in PostgreSQL with the cognitive flexibility of an LLM provides an effective balance between robustness and adaptability.
18. Signals for recruiters
This section summarizes the specific technical competencies demonstrated by the design and deployment of this architecture, providing a practical reference for engineering capabilities:
- Automation workflow design with n8n: Advanced orchestration of complex asynchronous data flows, retry handling, concurrency control, and balanced resource consumption in self-hosted systems.
- Supabase/PostgreSQL integration for tracking and traceability: Robust relational data modeling, definition of data-integrity constraints (unique constraints, check constraints), creation of performance indexes, and structured management of state machines at the persistence layer.
- Responsible scraping and data normalization: Ethical web-data extraction, rate-limit mitigation, active avoidance of third-party server overload, and parsing of unstructured sources into strict JSON schemas.
- AI for classification, scoring, and assisted resume generation: Integration of the OpenAI API using advanced Structured Outputs capabilities to interact deterministically with large language models while preventing type-related failures through clean data abstractions.
- Personal web-portfolio integrations: Development of Supabase Edge Functions written in TypeScript/Deno to dynamically consume complex payloads and control headless document-rendering engines through structured CSS rules.
- Human-in-the-loop systems with Telegram and manual review: Development of asynchronous conversational micro-interfaces based on webhooks and event-driven architectures in which the human component acts as the critical quality-validation gate to ensure ethical and consistent processes.