Document-driven interface testing tool

Document-driven interface testing tool

Quick start

Install HTTE command line

npm i htte-cli -g
 

Write test

Write configuration config.yaml

modules:
- echo 

Write test modules/echo.yaml

- describe: echo get
  req:
    url: https://postman-echo.com/get
    query:
      foo1: bar1
      foo2: bar2
  res:
    body:
      args:
        foo1: bar1
        foo2: bar2
      headers: !@exist object
      url: https://postman-echo.com/get?foo1=bar1&foo2=bar2
- describe: echo post
  req:
    url: https://postman-echo.com/post
    method: post
    body:
      foo1: bar1
      foo2: bar2
  res:
    body: !@object
      json:
        foo1: bar1
        foo2: bar2 

Run test

htte config.yaml
 

Results of the

   echo get (1s)
   echo post (1s)

2 passed (2s)
 

Original intention

Why do interfaces need to be tested?

  • Improve service quality and reduce bugs
  • Locate bugs earlier, saving debugging and processing time
  • Easier code changes and refactoring
  • Testing is also a document, which helps to familiarize yourself with service functions and logic
  • Service acceptance criteria

There are many projects but there is no interface test, because the test is difficult, the difficulty is:

  • Writing tests doubles the workload
  • Writing test code requires a certain learning cost
  • Data coupling between interfaces makes it difficult to write tests
  • Constructing the request data and verifying the response data itself is very boring and tedious
  • Test code is also code, and it will be corrupted if you don t spend effort to optimize iteration

Is there a strategy that not only allows us to enjoy the benefits of testing, but also to minimize its costs?

The answer after research is document-driven.

Describe the test in a document, and execute the document with a tool.

This is the original intention of HTTE.

Document-driven advantages

Easier to read

Such an interface has: its service address http://localhost:3000/add, use POSTand use jsonas a data exchange format, the data format of the request { a: number, b: number}and returns the data format {c: number}, function is achieved a, bsummed and returns the result c.

For this interface, the test idea: Pass data to this interface {"a":3,"b":4}and expect it to return{"c":7}

This test is written in HTTE in the form of a document.

- describe:  
  req:
    url: http://localhost:3000/add
    method: post
    headers:
      Content-Type: application/json
    body:
      a: 3
      b: 4
  res:
    body:
      c: 7 

List the request and response directly, and add a description to explain what the test is for, and a complete test is written.

Easier to read

Please take a look at the following two tests to guess what function the target interface implements.

- describe:  
  name: fooLogin
  req:
    url:/login
    method: post
    body:
      email: foo@example.com
      password: '123456'
  res:
    body:
      token: !@exist string
- describe:  
  req:
    url:/user
    method: put
      Authorization: !$conat [Bearer, ' ', !$query fooLogin.res.body.token]
    body:
      nickname: bar
  res:
    body:
      msg: ok 

Although you may not understand now !@exist, !$concat, !$querybut we should be able to understand the functions of these two rough interfaces, request response data format.

Since the test logic is carried by the document, HTTE easily obtains some of the coveted advantages of other frameworks:

Programming language independence

There is no need for the care backend to be implemented in that language, and no longer worry about switching from one language to another, let alone switching from one framework to another.

Low skill requirements, quick to get started

Pure documentation, no need to understand the back-end technology stack, or even programming. Novice employees and even clerical staff can quickly master and write.

High efficiency and fast development

It's easy to write, easy to read, low skill requirements, and of course it's fast to write. Finally, I can freely enjoy the advantages brought by the test, and avoid the trouble caused by the test to the greatest extent.

Naturally suitable for test-driven development

The documentation is fast and easy to write, and it is easy to adopt TDD development strategies. Finally, you can enjoy the advantages of TDD without side effects.

Instructions for use as a front-end interface

What should I do if I have swagger/blueprint documentation but still can t use the interface? Throw him the test document, it is full of examples.

As a back-end requirement document/development guidance document

New recruits or junior engineers may not be so familiar with the business, and their skills may not be so proficient. They need a longer learning period or adaptation period, and the quality of the written interface may not be enough. Having such a test document can greatly shorten this time and improve the quality of the interface.

HTTE advantages

In addition to all the advantages of document-driven testing, HTTE also has the following advantages.

Use YAML language

Not introducing a new DSL, but directly adopting YAML. There is no additional learning cost, it is easier to get started, and you can enjoy the existing YAML tools and ecology.

Use plugins to flexibly generate request verification responses

Let me talk about why plugins are needed in document-driven testing.

A certain interface has duplicate name detection, so we need to generate a random string when testing. How to describe random numbers in the document? An interface returns an expiration time. We need to verify that this time is 24 hours after the current time. How to define this time in the document?

The biggest obstacle to the document-driven testing strategy is that the document cannot carry complex logic and lacks flexibility. It is difficult to describe the probability of random strings and current time. Only functions can provide this flexibility. Plug-ins provide functions for documents and provide this flexibility.

The plug-in is presented in the form of a YAML custom label.

There is such a piece of code

req:
  body: !$concat [a, b, c]
res:
  body: !@regexp/w{3} 

!$concatAnd !@regexpis YAML label, a type of data which is user-defined. In HTTE, it is actually a function. So the above code looks like this from HTTE.

{
  req: {
    body: function(ctx) {
      return (function(literal) {
          return literal.join('');
      })(['a', 'b', 'c'])
    }
  }
  res: {
    body: function(ctx, actual) {
      (function(literal) {
          let re = new Regexp(literal);
          if (!re.test(actual)) {
              ctx.throw(' ')
          }
      })('\w{3}')
    }
  }
} 

In general, documents have the advantages of being easy to read, write and easy to use, but they cannot carry complex logic and are not flexible enough; while functions/codes can provide this flexibility, but they also bring too much complexity. HTTE adopts YAML format and encapsulates functions into YAML tags, which harmonizes this contradiction, integrates each other's advantages to the greatest extent, and almost avoids each other's shortcomings. This is the biggest innovation of HTTE.

There are two main operations on data in interface testing, constructing a request and verifying a response. So there are two kinds of plug-ins in HTTE.

  • Constructor (resolver), used to construct data, label prefix !$
  • Comparer (differ), used to compare check data, label prefix !@

Plug-in set:

  • builtin -Contains some basic and commonly used plugins

Componentization, easy to expand

The HTTE architecture diagram is as follows:

Each component is an independent module, opposing to complete a specific task. So it can be easily replaced and easily extended.

Let's combine an example to introduce the specific execution process of a test unit in HTTE, so that everyone is familiar with the functions of each component.

Another test

- describe:
  req:
    body:
      v: !$randnum [3, 6]
  res:
    body:
      v:  !@compare
        op: gt
        value: 2 

In the Runnerafter loading all tags YAML expand a function insert as defined in pseudo code is as follows. At the same time Runnersends runUnitevent.

{
  req: {//Literal Req
    body: {
      v: function(ctx) {
        return (function(literal) {
          let [min, max] = literal
          return Math.random() * (max - min) + min;
        })([3, 6])
      }
    }
  },
  res: {//Expect Res
    body: {
      v: function(ctx, actual) {
        (function(literal) {
          let { op, value } = literal
          if (op === 'gt') fn = (v1, v2) => v1 > v2;
          if (fn(actual, literal)) return;
          ctx.throw('test fail');
        })({op: 'gt', value: 2})
      }
    }
  }
} 

RunnerWill be Literal Reqpassed to Resolver, Resolverwork is recursive traversal Reqof the function and execution, to give a pure data values. And pass to Client.

req: {
 //Resolved Req
  body: {
    v: 5;
  }
} 

ClientAfter receiving this data, construct a request, encode the data into a suitable format (if it is JSON, it Encoded Reqwill become {"v":5}), and send it to the back-end interface service. ClientAfter receiving the response returned by the back-end service, the data needs to be decoded first. Assuming that this interface is an echo service, and the returned data is {"v":5}( Raw Res) and is JSON, the Client data will be decoded as:

res: {
 //Decoded Res
  body: {
    v: 5;
  }
} 

DifferAt this time we will get from Runnerthe Expected Resand from Clientthe Decoded Res. Its job is to compare the two.

DifferTraverses Expected Reseach of values, one by one to Decoded Resbe compared. Any inequality will throw an error and mark the test as a failure. If both are values, judge whether they are congruent. If a function is encountered, the comparison function will be executed. The pseudo code is as follows:

(function(ctx, actual) {
    (function(literal) {
      let { op, value } = literal
      if (op === 'gt') fn = (v1, v2) => v1 > v2;
      if (fn(actual, literal)) return;
      ctx.throw('test fail');
    })({op: 'gt', value: 2})
  }
})(ctx, 5) 

If the comparison function does not throw an error, it means the test passed. RunnerAfter sending test results are received by the doneUnitevent, and perform the next test in the queue.

ReporterMonitor Runnerevents sent, generate reports, or print to a terminal, or generate an HTML report file.

The interface protocol is extensible, and currently supports HTTP/GRPC

The interface protocol is provided by the client extension.

  • htte -suitable for HTTP interface testing
  • grpc -suitable for GRPC interface testing

The report generator is extensible and currently supports CLI/HTML

  • cli -output to the command line
  • html -output test report as HTML file

Elegantly resolved interface data coupling

Data is coupled between interfaces. A common example is to log in and get the TOKEN before you have the authority to place an order, send it to Moments, etc.

Therefore, an interface test often needs to access the data of another test.

HTTE handles this problem through session + plug-in.

Let's illustrate with examples.

There is a login interface, which is like this.

- describe: tom login
  name: tomLogin # <---   Why?
  req:
    body:
      email: tom@gmail.com
      password: tom...
  res:
    body:
      token: !@exist string 

There is a user interface to modify the name, it is a privilege interfaces, there must be a Authorizationrequest to return the head and bring login tokento use.

- describe: tom update username to tem
  req:
    headers:
      Authorization: !$conat [Bearer, ' ', token?] # <---   TOKEN  
    body:
      username: tem 

Reveal the answer

      Authorization: !$conat [Bearer, ' ', !$query tomLogin.res.body.token] 

You can also tomLogin.req.body.emailget the mailbox value by tomLogin.req.body.passwordobtaining a password. Is it elegant?

How is this achieved?

HTTE in Runnerthe start, it will initialize the session. Each time a unit test is executed, the execution result will be recorded in the session, including request data, response data, time-consuming, test results, etc. These data are read-only and ctxare exposed to the plug-in function, so the plug-in can access the data of the tests it has executed before.

The same test, resis also able to reference reqdata of.

- describe: res ref data in req
  req:
    body: !$randstr
  res:
    body: !@query req.body 

Use macros to reduce repetitive writing

An interface often around by the plurality of unit testing, and the interface has some of the same attributes, get the HTTP interface, for example, have req.url, req.method, req.type, do not have to re-write every call again?

- decribe: add api condition 1
  req:
    url:/add
    method: put
    type: json
    body: v1...
- decribe: add api condition 2
  req:
    url:/add
    method: put
    type: json
    body: v2... 

Macros are introduced to solve this type of repetitive input problem. It is also very simple to use, definition + reference.

Define the macro in the project configuration.

defines:
  add: # <--  
    req:
      url:/add
      method: put
      type: json 

Use this interface anywhere, just like this

- describe: add api with macro
  includes: add # <--  
  req:
    body: v... 

Debug and develop

This feature is achieved by combining command line options. The two related command-line options are: --bailstop execution if any test fails; --continuecontinue the test from where it was interrupted last time.

Combining these two options allows us to reset and execute the problem interface countless times until the debugging is passed.

Configuration

Optional configuration items:

  • session: Specify the storage location of the persistent session file. Generally, you do not need to fill it in. HTTE will generate a unique temporary file storage session for the project in the operating system temporary folder.
  • modules: Provide a list of test module files. The order of the list corresponds to the order of execution. HTTE will be configured in the same directory as the file modulesdirectory of the current directory and find the corresponding module file.
  • clients: Configure client extensions
  • plugins: Configure the plugin
  • reporters: Configure the report generator extension
  • defines: Define macro

Minimum configuration:

modules:
- auth
- user/order 

In this case clients, plugins, reporterit will use a default value.

Fully configured:

session: mysession.json
modules:
- auth
- user/order
clients:
- name: http
  pkg: htte-client-http
  options:
    baseUrl: http://example.com/api
    timeout: 1000
reporters:
- name: cli
  pkg: htte-reporter-cli
  options:
    slow: 1000
plugins:
- name: ''
  pkg: htte-plugin-builtin
defines:
  login:
    req:
      method: post
      url:/users/login 

Configuration patches are used to respond to configuration changes under environmental differences. For example, when writing in a test environment, the interface address is http://localhost:3000/api; in a formal environment, it needs to be changed to https://example.com/api. It is best to configure the patch to achieve.

Define patch files, patch file naming rules... If the project configuration file name htte.yaml, patch name prod, then the patch file name htte.prod.yaml.

- op: replace
  path:/clients/0/options/baseUrl
  value: https://example.com/api 

Use the command line --patchoption is selected patch file. For example, if we want to quote htte.prod.yaml, enter--patch prod

The patch file specification jsonpatch.com .

Test unit/group

The test unit is the basic unit of HTTE.

  • describe: Describe the purpose of the test
  • name: Define the test name, convenient !$queryand !@queryreferences can be omitted
  • client: Define the use of client extensions, if the project has only one client, it can be omitted
  • includes: Quote macro, you can quote multiple
  • metadata: Meta tags, HTTE engine specific data
    • skip: Whether to skip this test
    • debug: True indicates the request and response data details printed when reporting
    • stop: True means to terminate subsequent operations after executing the test
  • req: Request, check and fill in the corresponding client extension file
  • res: Respond, check the corresponding client extension file to fill in

Sometimes a functional test requires the cooperation of multiple test units to complete. In order to express this combination/level relationship, HTTE introduces the concept of group.

  • describe: Describe the purpose of the test
  • defines: Define special macros in the group, the syntax is the same as the global macros in the configuration
  • units: Elements in the group
- descirbe: group
  defines:
    token:
      req:
        headers:
          Authorization: !$conat [Bearer, ' ', !$query u1.req.body.token] 
  units:
  - describe: sub group
    units:
      - descirbe: unit
        name: u1
        metadata:
          debug: true
        includes: login
        req:
          body:
            username: foo
            passwold: p123456
        res:
          body:
            token: !$exist
  - descibe: unit
    includes: [updateUser, token]
    req:
      body:
        username: bar 

license

MIT