This growing popularity has even pushed major players like Meta and OpenAI to launch dedicated apps for creating and sharing AI-generated videos, such as Sora and Meta AI.

Most users currently consume AI videos primarily for entertainment rather than for news or reliable information. In fact, the same report shows that users are generally less likely to trust content on social media when it is AI-generated.

Nonetheless, the trust in AI content may shift with the introduction of avatars, regardless of whether they are realistic or not. If the creator can validate that the information shared by the avatar is indeed correct, people will follow and support the page, even if it is not a real person talking.

In this article, I’ll guide you through two methods for creating avatar videos using Kling AI. The first method uses Kling’s Motion Control, which replicates the movements of a real person and applies them to a character or image you provide. The second method combines text, an image, a prompt, and audio to generate an avatar video.

We’ll start by generating videos directly from the platform’s console, then move on to building a workflow that automatically generates and publishes them to social media using the Kling AI API and Python.

Create your first avatar video

You can create a free account on Kling AI, but video generation on the platform comes with a cost. New users receive a limited number of credits after registering, which are usually only enough to generate images or very short video clips.

Kling offers two main pricing options: a monthly subscription or the ability to purchase credits separately. API access is priced differently and will be discussed in a later section.

I bought a few credits some months ago, but if you’re only starting, the Standard plan has more perks than simply acquiring credits. That’s because buying credits won’t remove the watermark in your videos, and $5 only gets you 330 credits, while with the Standard plan, you will get up to 660 per month.

⚠ NOTE: If you’re planning on using the API instead, skip the above pricing options. Otherwise, you’ll be paying for using the platform and also for the API.

Before picking a pricing plan, you can see if you have enough credits to make your first experiment with the Motion Control model under the AI Video Generator tab.

This option lets you upload a reference video along with an image, which Kling AI uses to replicate the video’s movements and expressions. The number of credits required depends on the length of the reference video. Make sure you start with a short clip.

In addition, you must provide a prompt that supplies contextual and behavioral guidance for the output. You can make your avatar more extroverted and smiley or more serious, depending on how you structured your prompt.

Here’s an example with an image I created using Nano Banana Pro on Gemini:

Kling Motion Control - Dad Half Dev

This is "Kling Motion Control - Dad Half Dev" by Marco Rodrigues on Vimeo, the home for high quality videos and the people who love them.

https://vimeo.com/

The Motion Control requires you to record yourself or have a video reference. The other approach is to create an avatar that, instead of taking a reference video, takes a text, an audio, an image, and a prompt. On Kling AI, you can do this by selecting Video and then Avatar.

On the console, you’ll find several avatar examples ready to be used, but that’s not as fun as creating your own!

The steps are similar to Motion Control, but you need to add the speech (text) and an audio. The longer the speech, the more credits will be consumed. Instead of the text, you can also upload an audio file.

In the next section, we will use the Avatar 2.0 to automatically create informative videos based on summaries from articles. For that, we will need the Kling AI API and the Python programming language.

Create an authentication token for the Kling AI API

You can use the same Kling account to access the API console.

The pricing plans for the API usage are different than the ones previously discussed. In our case, we’re going to use the pricing that is visible under the Functional Models, which includes the Avatar, Lyp-Sync, and other products.

These are the available pricing packages:

Once you’ve picked your preferred package, you can create an API key, which has an Access Key and a Secret Key.

Now, you still won’t be able to make an API request with the two, you must do a JWT verification. Here’s the Python script:

import time
import jwt

access_key = “AP4...” # fill access key
secret_key = “adJme...” # fill secret key

def encode_jwt_token(ak, sk):
    headers = {
        “alg”: “HS256”,
        “typ”: “JWT”
    }
    payload = {
        “iss”: access_key,
        “exp”: int(time.time()) + 1800, # The valid time, in this example, represents the current time+1800s(30min)
        “nbf”: int(time.time()) - 5 # The time when it starts to take effect, in this example, represents the current time minus 5s
    }
    token = jwt.encode(payload, secret_key, headers=headers)
    return token

authorization = encode_jwt_token(access_key, secret_key)
print(authorization) # Printing the generated the authentication token

You can tweak the script to increase the expiration time. The output is a Bearer token that will be used for the API requests.

You can learn more about this and the following steps by reading the official API documentation here.

Create the audio and the avatar with the Kling AI API

Unlike the platform interface, where an avatar video can be created in just a few steps, using the API requires a multi-step workflow. You must first generate an authentication token (as explained earlier), then create the audio file, generate the video, and finally retrieve the video URL.

Let’s start by creating the audio file. In this case, we will choose a Voice ID from the spreadsheet provided by Kling AI.

import requests

AUDIO_URL = “https://api-singapore.klingai.com/v1/audio/tts”
API_TOKEN = “eyJhbGciOiJIUzI1Ni...”


headers = {
    “Authorization”: f”Bearer {API_TOKEN}”,
    “Content-Type”: “application/json”
}
payload = {
    “text”: (
        “Forget small steps—Ethereum is preparing for a massive leap. “
        “By 2026, the network is set to start scaling exponentially “
        “thanks to Zero-Knowledge (ZK) technology. Think of it as a total “
        “brain transplant for the blockchain, “
        “similar in scale to ‘The Merge’.”
        “Instead of every validator re-doing every single transaction, “
        “they’ll soon just verify tiny ZK-proofs. “
        “This shift could eventually skyrocket Ethereum from 30 “
        “transactions per second to a staggering 10,000. “
        “The best part? It’s so efficient you could theoretically “
        “run a node on a smartphone. While it’s a multi-year journey, “
        “2026 is when the real speed revolution begins.”),
    “voice_id”: “chat_0407_5-1”,
    “voice_language”: “en”,
    “voice_speed”: 1.2,

}

response = requests.post(AUDIO_URL, headers=headers, json=payload)
print(response.json())

The code snippet above takes a text from a Cointelegraph article as input, along with the voice_id, the voice_language and the voice_speed. The output is the URL for the voice file and the audio id. The response looks like this:

{’code’: 0, ‘message’: ‘SUCCEED’, ‘request_id’: ‘f14a385d-20b8-45a8-8469-b00d2aaded65’, ‘data’: {’task_id’: ‘835047569842642979’, ‘task_status’: ‘succeed’, ‘task_result’: {’audios’: [{’id’: ‘835047569859411990’, ‘url’: ‘https://v16-kling-fdl.klingai.com/bs2/klingai-kling-input-stunt-sgp/muse/834731746867421228/AUDIO/20251231/d147f6fa9948f358602fa42cd956bb21-f9dc5b57-6d5a-4e0b-8f37-fe06572484b0?cacheKey=ChtzZWN1cml0eS5rbGluZy5tZXRhX2VuY3J5cHQSsAFGHCN_CgjsePaPU4pF4kLs3wTWpJBrE6KgbqzxUmj753zYHmfUr372mu1-CGXxj0PJ0GTW-XLYGUjUYGHx2uENIrR53Ud_K8H3JnHMj-1YFkqx8dEVNnvpTVqylkCUqCEujdgRNETYMqctNaJdWzg3ymVJGWPQ-Ur5QKyoFPpmWzP9If7YAeN3pRXlpTYjIZNM68E_kZeGIzUAMBv2Ml60ebpaIkk9yuBeMvh8DLZBbxoSOFvh80_9xnI_XufDmu_AA0PdIiA1c1FjVWNODooKSYUEUJ6KO8OkJ3eiwBnzMN53F6ne9igFMAE&x-kcdn-pid=112781&ksSecret=7ba1734ea6d1d4e041b846623705b975&ksTime=697ba710’, ‘duration’: ‘39.384’}]}}}

Now we’re going to use the audio id and an image to generate the video.

This is the avatar image I used in my video:

Here’s the script to generate the video:

import requests
import base64

with open(”character_female.png”, “rb”) as image_file:
    image_data = image_file.read()
    img_base64 = base64.b64encode(image_data).decode(”utf-8”)

headers = {
    “Authorization”: f”Bearer {API_TOKEN}”,
    “Content-Type”: “application/json”
}

payload = {
    “image”: img_base64,
    “audio_id”: 835047569859411990,
    “mode”: “std”,
    “prompt”: (
        “Create a realistic talking avatar with a natural smile and focused”
        “, confident expression, speaking directly to the camera.”
        “\n\nCamera & Background: Keep the camera fixed and do not move or “
        “alter the background in any way.\n\n”“Body Movement:\n”
        “Add subtle, natural upper-body movements (slight torso “
        “shifts, gentle head tilts, minimal shoulder motion) to avoid a “
        “static or robotic appearance.\n\n”
        “Performance style: Calm, engaging, and professional—natural speech “
        “rhythm, smooth facial animations, and realistic eye and “
        “mouth movements.\n\n”
        “Overall quality: Photorealistic lighting, stable framing, “
        “and lifelike motion without exaggeration.”)
}

response = requests.post(
    “https://api-singapore.klingai.com/v1/videos/avatar/image2video”,
    headers=headers,
    json=payload)


print(response.json())

After running this script, the video may take some time to become available, depending on the audio length. Be sure to add a delay before proceeding to the fetch step. This snippet produces the following output:

{’code’: 0, ‘message’: ‘SUCCEED’, ‘request_id’: ‘d6b40d68-6866-4b31-849e-a301036d0b14’, ‘data’: {’task_id’: ‘835047984131473475’, ‘task_status’: ‘submitted’, ‘created_at’: 1767119472638, ‘updated_at’: 1767119472638}}

In the next and final step, we’re going to use the request_id and task_id to obtain the video URL. Here’s how:

request_id = “d6b40d68-6866-4b31-849e-a301036d0b14”
task_id = ‘835047984131473475’

url = f”https://api-singapore.klingai.com/v1/videos/avatar/image2video/{request_id}”


headers = {
    “Authorization”: f”Bearer {API_TOKEN}”,
    “Content-Type”: “application/json”
}

payload = {
    “task_id”: task_id,
}

response = requests.get(
    url,
    headers=headers,
    json=payload)


print(response.json())

The response has the video URL from where you can download the video:

{’code’: 0, ‘message’: ‘SUCCEED’, ‘request_id’: ‘151d0403-a017-42af-89af-668d0c2ff53a’, ‘data’: {’task_id’: ‘835047984131473475’, ‘task_status’: ‘succeed’, ‘task_info’: {}, ‘task_result’: {’videos’: [{’id’: ‘835047984186015791’, ‘url’: ‘https://v16-kling-fdl.klingai.com/bs2/upload-ylab-stunt-sgp/muse/834731746867421228/VIDEO/20251231/7bc028554e69b8b6c9527496bdfc00e6-901b7d5a-40c3-4f54-b4da-d499cfe599d7.mp4?cacheKey=ChtzZWN1cml0eS5rbGluZy5tZXRhX2VuY3J5cHQSsAH1weioZR4lXm4IjZ4hSNdgLYXSg-qYje0OYPNG_RCV-L7ISZHkFYzkwOvZ-Q2YpAt514WqvbwsB7Y648-4JFYOg0pQV_9rn2AL6UgS2MPLQDcnjSwhk4URxBFL9kKceOwExCp7a2LwoJ7MSjYgJn1HOi661_f17mkhyCzCGmGQcjsrM7x8-7IUb6iLMJ95L8jrGaln6YfxLxwRLHwHOzK2MrP334DYiQaJW7p_KrmcDhoSv3Uc89R3_xETrF65D8P8-8MOIiD_d69m8PnwyzUGrpf3N1pm6gCX1XJxFgrwpIp9kHIlCigFMAE&x-kcdn-pid=112781&ksSecret=29b1f4a65600c88cac188b609446c9bc&ksTime=697baad9’, ‘duration’: ‘39.466’}]}, ‘task_status_msg’: ‘’, ‘created_at’: 1767119472638, ‘updated_at’: 1767120346646}}

Here’s the final output:

Kling AI Standard Avatar Video

This is "Kling AI Standard Avatar Video" by Marco Rodrigues on Vimeo, the home for high quality videos and the people who love them.

https://vimeo.com/

This video alone consumed around 16% of the total points available (I’m using the cheapest API pricing plan). This means that with $9.79, I can generate up to six videos, if they all have the same length.

Automate posts on social media

If you want to automate social media posts on platforms like X, Bluesky, and Instagram, you can use either cron jobs on your local machine or a remote VPS.

I prefer running automation and web scraping on a Contabo VPS, using my local machine only for testing. This keeps my main workstation free and avoids unnecessary RAM and resource usage.

I’m currently using the Cloud VPS 30 for only €11.2/month, which has 24 GB RAM (more than my laptop), 8 vCPU Cores, and 400 GB SSD. This plan is more than enough for most automation tasks, but feel free to have a look at the other VPS and VDS options available.

When it comes to social media posting, I’ve previously written two articles. One for X (former Twitter) and another for BlueSky.

Create a bot to publish on X

Learn How to Build a Bot on X Step-by-Step With This Tutorial

Instead of using the X API V2 documentation, make a bot on steroids with these steps.

https://blog.dadhalfdev.com

Create a bot to publish on BlueSky

Easy Steps to Automate Your Posts on BlueSky with Python

Use the BlueSky Python SDK and some creativity to automate posts on BlueSky for free.

https://blog.dadhalfdev.com

If you want to build an avatar like mine that talks about crypto, tech, and world news, you’ll need a web scraper to extract fresh daily content. This content can then be summarized and used as the audio script for your avatar.

If you’re not familiar with summarizing content using LLMs and Python, take a look at the article above on building bots for X.

As for the web scraper, if you’re short on ideas, you can use one of mine:

• Aljazeera Scraper: Extract articles from Aljazeera, providing essential insights into current events and regional tensions.

• TechCrunch Scraper: Get the latest tech and AI news from TechCrunch.

• Cointelegraph Scraper: Extract data from Cointelegraph, a leading platform for cryptocurrency and blockchain-related news.

You can find many other scrapers available on Apify.

Do you need help with automation, web scraping, AI, data, or anything that my laptop can deliver? Feel free to reach me on Upwork! 👨‍💻

Conclusion

The costs associated with creating a single avatar video can quickly add up. For example, with less than $10, I would only be able to publish six videos in total.

That said, this expense is relatively small when compared to the time and effort required to produce a traditional social media video. Recording myself talking about the latest crypto news would involve multiple takes, video editing, and careful audio adjustments.

All of these manual steps are far more costly than a couple of dollars per video.

An avatar that consistently generates content can instead be used to drive traffic to your website, support affiliate marketing, or power other growth strategies.

As we’ve seen at the beginning of this article, users are consuming more AI-generated videos than ever. The real challenge is creating an engaging, value-driven avatar. One that can generate passive income while you sleep.

Curious about what I’m building? 👀

A key element in this is Artificial Intelligence (AI), which requires vast amounts of data for model training and therefore relies on massive crawlers running nonstop. At the same time, AI has also made web scraping more accessible, allowing non-developers to collect data through vibe coding and seamless APIs.

According to a report by Market Research Future, the web scraper software market was valued $1.01 billion in 2024. In 2025, the Mordor Intelligence report shows that the same market is now valued at $1.03 billion, a steady 1.98% increase in just one year. The same report projects that the web scraping market will reach $2 billion by 2030, nearly 98% growth in just six years.

While web scraping is more accessible with the integration of AI, many websites have also shielded themselves against massive bot requests. Making the job difficult for many APIs and web scrapers powered by vibe coding scripts.

In this piece, I’m sharing with you 5 tips that Large Language Models (LLMs) don’t tell you to build a robust web scraper that can outperform 90% of the websites with only one tool: Playwright.

First steps with Playwright

At the time I’m writing, I have over 20 Apify actors running, all built using Playwright.

Before, I used to web scrape with Selenium, Beautiful Soup, and Scrapy, but I’ve realized that I can have successful bots only using Playwright, so why complicate things?

Playwright is a headless browser considered to be faster than its competitors and with better support for modern websites. Documentation is easy to follow and supports multiple languages.

To install Playwright, you need the following pip command:

To install the supported browsers, use this command in your terminal:

Now, the basic component of Playwright is often called page and it is instantiated this way:

from playwright.async_api import async_playwright


async with async_playwright() as pw:

    # creates an instance of the Chromium browser and launches it
    browser = await pw.chromium.launch(headless=False)

    # creates a new browser page (tab) within the browser instance
    page = await browser.new_page()

I will not cover all of Playwright’s functions in this piece, but I will highlight the ones I use in most of my crawlers and have proven to be indispensable:

# Go to page and wait to render full content: images, APIs and more
await page.goto(”some_url”, wait_until=”networkidle”)

# Go to page and wait only for DOM structure (initial HTML). Much faster.
await page.goto(”some_url”, wait_until=”domcontentloaded”)

# Create some timing between actions
await page.wait_for_timeout(3000)

# Get text content only
await page.locator(”a”).text_content()

# Get an attribute
await page.locator(”a”).get_attribute(”href”)

# Get all elements with the same class
elements = await page.locator(”a.font-big”).all()

# Click the first element
await page.locator(”a.font-big”).first.click()

# Click the last element
await page.locator(”a.font-big”).last.click()

# Click element by index
await page.locator(”a.font-big”).nth(3).click()

In addition to these examples, it is also useful to use page.get_by_role(), page.get_by_title(), page.get_by_label(), and so on, whenever you can. This is usually more straightforward than using the page.locator().

For more about Playwright’s functions, visit the well-structured official documentation.

Now we will have a look at the 5 tips to improve your web scraping skills while integrating Playwright. They aim at answering the following questions:

1. How to increase the stealth performance?

2. How to intercept the website’s HTTP response?

3. How to handle infinite scroll websites?

4. How to target HTML elements accurately?

5. How to make proper proxy rotation?

Let’s get started!

1 - How to increase the stealth performance?

If you’re familiar with Playwright, you’ve probably heard of playwright_stealth.

While it can work for some websites, it is definitely an outdated solution to improve your stealth performance.

The alternative: Camoufox.

This open-source anti-detect browser can be integrated with the Playwright API. It works by providing Camoufox with detailed device characteristics, including the operating system, device information, navigator properties, headers, screen size, and more.

To install Camoufox, use this pip command:

pip install -U camoufox

Your playwright’s page object can be instantiated in the following way:

from camoufox import AsyncCamoufox

async with AsyncCamoufox(
    os=”macos”,
    humanize=True,  # Enable humanized cursor movement
    headless=False,  # Keep visible for debugging
    window=(1280, 720),  # Set window size
) as browser:

  page = await browser.new_page()

Using this browser won’t solve all your issues. You may still get blocked by websites protected by Datadome, but it is significantly better than using Playwright only.

You can also add proxies, tweak the parameters, and add BrowserForge to see if the stealth is improved.

2 - How to intercept the website’s HTTP response?

Many developers who engage in web scraping tend to strictly extract and parse the HTML content. While this solution works, it may not be the fastest way to crawl data, and you may miss important metadata and features.

Many modern websites don’t load all their data in the initial HTML. Instead, they fetch data dynamically from APIs (such as JSON and GraphQL) after the page loads.

Therefore, before tackling the HTML content, check if the website has an API response. This can be triggered by simply loading the page, clicking on buttons, or scrolling the page.

Let’s take the Hackernoon website as an example.

In this case, there’s a JSON API response for each article that contains a lot of useful information about it, including the title, the slug, the ID, the author’s name, the image, and even the article’s body.

To extract the information from the JSON, you need to place this code before the event is triggered:

# Create set to track processed article IDs
processed_ids = set()

# Get response and set up response listener
response_received = asyncio.Event()
page.on(
    “response”,
    lambda response: check_json(
        response, response_received, processed_ids))

Now you can create a function to save the content in JSON format or CSV:

async def check_json(response, response_received, processed_ids):
    “”“ Get data from HackerNoon JSON response “”“

    url_response = (”https://hackernoon.com/_next/data/”)

    if url_response in response.url:
        try:
            items = await response.json()
            article_data = items[’pageProps’][’data’]
        except Exception as e:
            return

        if article_data:
            response_received.set()

            # Extract article features based on example.json structure
            try:
                article_id = article_data.get(’id’)
                
                # Check for duplicates using article ID
                if article_id in processed_ids:
                    return
                
                # Add to processed IDs set
                processed_ids.add(article_id)
                
                profile = article_data.get(’profile’, {})
                
                # Extract stats data
                stats = article_data.get(’stats’, {})
                
                post_data = {
                    ‘id’: article_id,
                    ‘title’: article_data.get(’title’),
                    ‘slug’: article_data.get(’slug’),
                    ‘link’: “https://hackernoon.com/” + \
                        article_data.get(’slug’),
                    ‘excerpt’: article_data.get(’excerpt’),
                    ‘tldr’: article_data.get(’tldr’),
                    ‘articleBody’: article_data.get(’articleBody’),
                    ‘createdAt’: article_data.get(’createdAt’),
                    ‘mainImage’: article_data.get(’mainImage’),
                    ‘mainImageHeight’: article_data.get(’mainImageHeight’),
                    ‘mainImageWidth’: article_data.get(’mainImageWidth’),
                    ‘socialPreviewImage’: article_data.get(
                        ‘socialPreviewImage’),
                    ‘parentCategory’: article_data.get(’parentCategory’),
                    ‘tags’: article_data.get(’tags’, []),
                    ‘commentsCount’: article_data.get(’commentsCount’),
                    ‘pageViews’: stats.get(’pageviews’),
                    ‘arweave’: article_data.get(’arweave’),
                    ‘author_name’: profile.get(’displayName’),
                    ‘author_handle’: profile.get(’handle’),
                    ‘author_avatar’: profile.get(’avatar’),
                    ‘author_bio’: profile.get(’bio’),
                    ‘author_isBrand’: profile.get(’isBrand’),
                    ‘author_isTrusted’: profile.get(’isTrusted’)
                }

                # Save “post_data “data to JSON or CSV
                    
            except Exception as e:
                print(”Error extracting article data: %s”, str(e))   

Where do you see the URL response? You can find it in Headers under Network by inspecting the website’s page.

The Response tab under Network displays the JSON structure with all the fields you can scrape.

Now you know, next time, check this before engaging in intense HTML parsing, and note that the APIs can be triggered in different ways. Make sure to try buttons, filters, pagination, and scrolling.

This is how I built my Hackernoon Apify actor. You can try it yourself here.

3 - How to handle infinite scroll websites?

Many modern websites use infinite scroll to improve loading speed and improve the user experience. But they are definitely not loved by web scrapers.

Dealing with infinite scrolling can be challenging because websites tend to use different strategies to load their content. Scrolls can be located on different sides of the page. Some pages might not load content if you don’t move to the exact place on the page, others might require clicking to load more content, and so on.

The simplest way to scroll a page with Playwright is by using the wheel function:

n_scrolls = 10
for i in range(n_scrolls):
      await page.mouse.wheel(0, 15000)
      print(’scrolling’)
      await page.wait_for_timeout(3000)
      await page.mouse.wheel(0, -1500)
      print(’move up’)
      await page.wait_for_timeout(3000)

The reason why I’m using a loop in the example above is that most pages with infinite scroll take some time to load the hidden content. Therefore, it may not work if you just scroll to the bottom of the page. With the loop, you can ensure the new data is loaded in each iteration.

While this strategy works, sometimes you can’t trigger the loading this way. The best approach is to move the cursor to the nearest element that triggers it. In the case of a news website, most likely the last visible element is an article/card, so that’s a good reference to scroll down to.

Let’s see how to implement it.

# scroll to the item
location = page.locator(”article”).last
await location.scroll_into_view_if_needed()

With the two code snippets above, you should be able to handle most of the infinite scroll websites. Note that the more you scroll, the harder it gets to load, and your scraper can get very slow.

4 - How to target HTML elements accurately?

Sometimes using the page.locator() can be a real frustration. You try several classes, attributes, and text, and still the script prompts an error.

For those cases, my first approach is to target the main classes. Meaning that if you have an element with a lot of classes, start by tackling it with only one or two classes. This will make Playwright prompt an error similar to this:

      playwright._impl._errors.Error: Locator.text_content: Error: strict mode violation: locator(”div.container-Qnseki”).first.locator(”a.pencraft.pc-reset”) resolved to 2 elements:
          1) <a data-testid=”post-preview-title” href=”https://newsletter.theaiedge.io/p/adalflow-a-pytorch-like-framework” class=”pencraft pc-reset color-pub-primary-text-NyXPlw font-pub-headings-FE5byy clamp-y7pNm8 clamp-3-lxFDfR reset-IxiVJZ”>AdalFlow: A PyTorch-Like Framework to Auto-Optimi…</a> aka get_by_role(”link”, name=”AdalFlow: A PyTorch-Like”)
          2) <a href=”https://newsletter.theaiedge.io/p/adalflow-a-pytorch-like-framework” class=”pencraft pc-reset color-primary-zABazT line-height-20-t4M0El font-text-qe4AeH size-15-Psle70 clamp-y7pNm8 clamp-2-kM02pu reset-IxiVJZ”>AI Agent frameworks are becoming just as importan…</a> aka get_by_role(”link”, name=”AI Agent frameworks are”)

Since it doesn’t know which element you are referring to, it will kindly present you with multiple options.

From those options, you can use the recommended Playwright function to grab it, or use nth() with the right index.

If your selector still doesn’t work, switching to XPath inside the page.locator() function can often deliver more accurate results.

This is because XPath allows you to navigate the structure of the DOM directly.

Let’s take a look at this example.

Suppose you’re trying to target a button with a dynamic class name that changes on every page load. A CSS selector like this might fail:

button = page.locator(”.dynamic-button-class”)

Instead, you can use an XPath expression that targets the element by its text content or position:

button = page.locator(”//button[contains(text(), ‘Submit’)]”)

There are also Playwright built-in functions that mimic XPath:

button = page.locator(”button”).filter(has_text=”Submit”)

You can either decide to learn the XPath expression language or explore more Playwright’s documentation. But avoid very long XPath expressions to optimize code readability.

5 - How to make proper proxy rotation?

Some of you are familiar with proxies and proxy rotation, but the real challenge arises when choosing the right proxy for your needs.

If anonymity and avoiding raising flags are your top priorities, residential proxies are the best option. However, if you are web scraping a website that doesn’t have strong blocking measures, you may want to prioritize speed, and datacenter proxies are a better solution.

I rely on proxy rotation when scraping more challenging websites, and IPRoyal’s trustworthy residential proxies have consistently met my needs.

IPRoyal proxies are also one of the cheapest in the market, with an excellent score on review platforms such as G2 and Trustpilot. You can create an account in a few seconds, without requiring too much personal information, and you can even pay with cryptocurrencies, if you don’t want to have your credit card attached to it.

Once logged in, you have access to a user-friendly dashboard where you can select the best proxy option for your needs. If you don’t know what works best, you can answer a short survey to help you out.

If you decide to use residential proxies like me, you’ll see a page with your username, password, hostname, port, and several filters that allow you to customize your proxy’s location.

To implement proxy rotation, I select up to 7 different countries and cities, and I add them to a Python list in my script. It looks like this with Camoufox:

proxy_list = [
    None,  # No proxy first
    “<key>_country-fr_city-toulouse”,
    “<key>_country-es_city-valencia”,
    “<key>_country-pt_city-lisbon”,
    “<key>_country-es_city-madrid”,
    “<key>_country-es_city-huelva”,
    “<key>_country-es_city-sevilla”,
]

for proxy_element in proxy_list:
    # Configure browser settings based on whether using proxy or not
    browser_config = {
        “os”: “windows”,
        “humanize”: False,  # Enable humanized cursor movement
        “headless”: False,  # Keep visible for debugging
        “window”: (1280, 720),  # Set window size
    }
    
    # Add proxy configuration if proxy_element is not None
    if proxy_element is not None:
        browser_config.update({
            “geoip”: True,
            “proxy”: {
                “server”: “geo.iproyal.com:11201”,
                “username”: “<my_username>”,
                “password”: proxy_element
            },
        })
    
    async with AsyncCamoufox(**browser_config) as browser:
      # If page is accessed continue the script, if not try another proxy 

      

While proxy rotation is beneficial to bypass anti-bot measures, it slows down your web scraper’s speed. Therefore, if speed is a top requirement, you can rent a Virtual Private Server (VPS), so you can run your script with multiple instances and at the same time, one for each proxy.

I use Contabo’s Cloud VPS 30 for only €12.5/month, which has 24 GB RAM (more than my laptop), 8 vCPU Cores, and 400 GB SSD. This plan is more than enough for most automation tasks, but feel free to have a look at the other VPS options available.

With my Contabo VPS, I can run concurrent web scrapers and schedule them using cron jobs at different times. This approach helps prevent my IP from being flagged while also conserving RAM for my primary tasks.

Do you need help with automation, web scraping, AI, data, or anything that my laptop can deliver? Feel free to reach me on Upwork! 👨‍💻

Conclusion

In this piece, we’ve covered the importance of Playwright headless browser for web scraping, and how it serves most of the web scraping tasks. Showing that you don’t need to learn multiple web scraping frameworks to build successful and performant crawlers.

Then we highlighted 5 different tips that vibe coding tools don’t tell you, but they make all the difference to turn you from a beginner to an advanced web scraping developer.

First, we discussed how to increase stealth and avoid being blocked by integrating Camoufox with Playwright.

Next, we looked at how intercepting HTTP responses before parsing HTML can save valuable time and help you capture hidden or critical data that might otherwise be missed.

We then examined how to handle infinite scroll pages efficiently using Playwright’s built-in functions, followed by strategies for optimizing data extraction with XPath or Playwright’s native selector capabilities.

Finally, we explored the differences between datacenter and residential proxies, highlighted the best options to start with, and explained how to integrate them effortlessly with Camoufox.

As demand for web scraping continues to grow, you can start applying these tips to land freelance projects on platforms like Upwork, build and monetize Apify actors for passive income, or simply create efficient crawlers for your own projects.

Perhaps this article will contribute to increasing even more the percentage of bots in internet traffic 🤖

Curious about what I’m building? 👀