Artillery

Install Artillery

npm install -g artillery

To check that the installation succeeded, run:

artillery -V

You can try artillery dino too.

Run a quick test

artillery quick --count 10 -n 20 https://artillery.io/

artillery quick is used for ad-hoc testing. Above command will create 10 virutal users each of which will send 20 HTTP requests to https://artillery.io/. use -k to ignore certificate errors. -o .json to extract test results in JSON file.

Run a test script

You write your load testing scripts and tell Artillery to run them. Scripts are written in YAML, with the option to write JavaScript to write custom logic.

Run a test script and generate report

artillery run -o <report_name>.json <test_file>.yml

Convert the above generated report to HTML

artillery report <report_name>.json

Putting A Test Script Together

load testing scripts have two main parts - config and scenarios

config section

specify target (such as address of the API server under test)
load progression (telling artillery for eg. to create 20 virtual users every second for 10 minutes)
can set a variety of other options such as HTTP timeout settings, or TLS settings

scenarios section

here you define what virtual users created by Artillery will be doing

Copy following code into a test.yml file:

config:
  target: "https://staging1.local"
  phases:
    - duration: 60
      arrivalRate: 5
    - duration: 120
      arrivalRate: 5
      rampTo: 50
    - duration: 600
      arrivalRate: 50
  payload:
    path: "keywords.csv"
    fields:
      - "keywords"
scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          body: "kw={{ keywords }}"
          capture:
            json: "$.results[0].id"
            as: "id"
      - get:
          url: "/details/{{ id }}"
      - think: 3
      - post:
          url: "/cart"
          json:
            productId: "{{ id }}"

Running the test

artillery run test.yml

Test report will look similar to this:

Complete report @ 2019-01-02T17:32:36.653Z
  Scenarios launched:  300
  Scenarios completed: 300
  Requests completed:  600
  RPS sent: 18.86
  Request latency:
    min: 52.1
    max: 11005.7
    median: 408.2
    p95: 1727.4
    p99: 3144
  Scenario counts:
    0: 300 (100%)
  Codes:
    200: 300
    302: 300

Scenarios launched is the number of virtual users created
Scenarios completed is the number of virtual users that completed their scenarios
Requests completed is the number of HTTP requests and responses or WebSocket messages sent
RPS sent is the average number of requests per second completed
Request latency is in milliseconds, and p95 and p99 values are the 95th and 99th percentile values (a request latency p99 value of 500ms means that 99 out of 100 requests took 500ms or less to complete)
Codes provides the breakdown of HTTP response codes received

If you see NaN (“not a number”) reported as a value, that means not enough responses have been received to calculate the percentile.

If there are any errors(such as socket timeouts), those will be printed under Errors in the report as well.

Testing HTTP

HTTP-specific configuration

TLS/SSL

By default, Artillery will reject SSL certificates that it’s unable to validate, e.g. self-signed certificates. You may see errors such as UNABLE_TO_GET_ISSUER_CERT_LOCALLY, UNABLE_TO_VERIFY_LEAF_SIGNATURE, CERT_UNTRUSTED or one of the other validation error codes when that happens.

You can disable certificate validation with either of the following two options

Pass -k (or --insecure) flat to artillery run or artillery quick
Set rejectUnauthorized: false in config.tls:
```
config:
  target: "https://myapp.staging:3002"
  tls:
    rejectUnauthorized: false
scenarios:
  - ...
```
Note: This option can be useful for testing in a development / staging environment, but should never be used when testing a production system.