Katarzyna Kmiotek

Does your team use Playwright for E2E tests? Are you working on performance tests and looking for a solution that will allow easy integration with current tooling? That was me!
While ago came across this article that talks about Artillery and Playwright integration.
The investigation of how to implement the above solution was great learning of Artillery tool that decided to document it briefly.

What is Artillery?

Artillery tests are written in YAML and test scenarios simulate virtual user (VU) journey.
Test config (either defined in a separate file or as part of the test) allows configuring the target URL, variables, environments and plugins. It allows also the loading of test data from CSV or loads JS scripts (processor).
In the config file, you can also define test phases (values that your VUs scale up to).
Important to remember about the config file: if something doesn't work but you provided all required values - it's probably indentation (classic YAML :) )

1config:
2  target: "https://api.demoblaze.com"
3  tls:
4    rejectUnauthorized: false
5  processor: '../processors/setup.js'
6  environments:
7    load:
8      phases:
9        - name: 'warming up'
10          duration: 300
11          arrivalRate: 10
12        - name: 'start'
13          duration: 300
14          arrivalRate: 20
15    functional:
16         #  single VU is used here
17        plugins:
18            expect: {}

Variables

You can define them in the config file or dynamically save them and use them in the test's context.
To use them within the scenario just import them with {{ }} .
Here is an example of how to use variables defined in the config, environment variables ($processEnvironment) and how to capture an API response assigned it as a variable and then use it. ($ is a symbol of the response object).

1scenarios:
2- name: 'add to the cart test'
3  flow:
4  # variable defined in the config
5  - log: "{{ token }}"
6  # env variable
7  - log: "{{ $processEnvironment.token }}"
8  - post:
9      json:
10        token: "{{ $processEnvironment.token }}"
11      url: /check
12      capture:
13        - json: "$"
14          as: response
15  - log: "{{ response }}"

Expect

With help of in-build, plugin Expect you can add assertions to test scripts for smoke tests execution on one VU.

1scenarios:
2- name: 'add to the cart test'
3  - post:
4      json:
5        token: "{{ $processEnvironment.token }}"
6      url: /check
7      expect:
8        - statusCode: 200
9        - contentType: json
10        - hasProperty: Item.expiration

result:

1* POST /check 
2  ok statusCode 200 
3  ok contentType json 
4  ok hasProperty Item.expiration 
5* POST /view 
6  ok statusCode 200 
7* OPTIONS /check 
8  ok statusCode 200 
9* OPTIONS /view 
10  ok statusCode 200

How to write YAML scenarios?

If you think writing your YAML scenarios will be time-consuming have a look at the Artillery-har-to-yaml project that allows you to convert the HAR file (so recorded browser interactions) to Artillery YAML scenario with just one command.
But how to get the HAR file?

• use DevTools in your browser, open the network tab, click preserve logs and click through the application as a user for the scenario you want to test. Then right-click somewhere in the Network tab and select save all as HAR with content
• use Playwright's most recent feature that allows recording a HAR file as part of the test. (docs)

What is Artillery Engine Playwright?

This is a collaboration between the browser provided ( Playwright ) and the engine to spin up hundreds of VU ( Artillery ). This way we can verify how a browser behaves under a huge user's load would FCP (First Contentful Paint) or LCP (Largest Contentful Paint) time change, would it affect the Time to Interactive score?
Artillery scenario would look like this:

1config:
2  target: "https://www.demoblaze.com/"
3  phases:
4    - arrivalRate: 1
5      duration: 10
6  engines:
7    playwright: 
8      launchOptions:
9        headless: false
10      contextOptions:
11      # logged in user
12        storageState: "./storageState.json"
13  processor: "../processors/addToCart.js"
14scenarios:
15  - name: "add to cart browser test"
16    engine: playwright
17    flowFunction: "addToCart"
18    flow: []

Here the target is the actual domain, not the backend service; the processor value points to the file where the method addToCart is defined.
The Playwright engine has access to the Page object so all methods are available on it. You also access to Artillery objects like userContext or events (allowing to emit custom metrics) I used npx playwright codegen <URL> command to generate addToCart function below.

1async function addToCart(page, context) {
2  await page.goto(context.vars.target);
3  await page.locator('a:has-text("Samsung galaxy s6")').click();
4  page.once('dialog', dialog => {
5    console.log(`Dialog message: ${dialog.message()}`);
6    dialog.dismiss().catch(() => {});
7  });
8  await page.locator('text=Add to cart').click();
9  await page.locator('#cartur').click();
10}

Using Playwright for set up of Artillery tests

You may not necessarily be testing a long user journey, maybe the test scenario doesn't need to have an e2e path, maybe just a load check of search functionality that the logged-in user can perform. You don't want to cover API calls (or UI steps) that cover the login part.
Artillery comes with handy hooks beforeScenario , afterScenario , beforeRequest and afterResponse where you point to the methods defined in the processor file.

1scenarios:
2- name: 'add to the cart test'
3  beforeScenario: setUserData
4  beforeRequest: setHeaders
5  flow:
6  - post:
7      json:
8        token: "{{ $processEnvironment.token }}"
9      url: /check

I use Playwright to perform login action (with API) get auth token, save it in context and use it in tests.

1const playwright = require('playwright');
2
3async function setUserData(context, ee, next) {
4  const apiContext = await playwright.request.newContext({
5    baseURL: context.vars.target,
6    ignoreHTTPSErrors: true,
7  });
8  const userLoginData = {
9    username: process.env.USER,
10    password: process.env.PASSWORD,
11  };
12  // get user data
13  const response = await apiContext.post('/login', {
14    data: userLoginData,
15  });
16  const loginResponse = await response.json();
17  // get a cookie, token, auth value from response
18  const tokenString = loginResponse.split(': ')[1];
19  // save user token value as variable in context used in tests
20  context.vars.$processEnvironment.token = tokenString
21
22  // if auth is stored in headers you can use an example below
23  // const headers = response.headersArray();
24  // const cookie = headers.find((obj) => obj.name === 'Set-Cookie').value;
25  // // save a cookie value
26  // context.vars.$processEnvironment.cookies = cookie;
27
28  // do the same with any other response attributes that can be useful to keep let's say user id 
29  next();
30}

I can also use setHeaders method as part of beforeRequest hook to set the cookies value of this token before each call executed as part of the scenario.

1async function setHeaders(requestParams, context, ee, next) {
2  requestParams.cookies =  { tokenp_: context.vars.$processEnvironment.token}
3  next();
4}

More about function signatures you can find here

And of course, you can find the code for this post in repo