Test fails when run manually but passes in collection tests

jesii7 · August 16, 2022, 4:37pm

Your question may already have an answer on the community forum. Please search for related topics, and then read through the guidelines before creating a new topic.

Here’s an outline with best practices for making your inquiry.

My question:
I’m creating my first tests around a graphQL collection. I started by creating two simple tests to ensure that a query executed successfuly and did not contain an “errors” element.

I tested this against a single query by modifying the query parameters to generate an error and ran the two tests – I got a test failure as expected.

When I run the entire collection using these same two tests, that particular query passes even though it failed when tested manually.

Details (like screenshots):
Here’s the individual query, with query parameters. It runs fine with valid parameters:

  query GET_CLIENT_NAMES($id: ID!, $asOfDate: ID!) {
    data: GetClient(id: $id, asOfDate: $asOfDate) {
      displayName
    }
  }
Query details changed for privacy
{
    "id": "XXXX",
    "asOfDate": 1653980400,
    "portfolioId": "XXXX:1653980400"
}

I’ve modified the parameters so that they are invalid and will generate an error.

These are the two query tests:

pm.test("Ensure 200 status code", function() {
    pm.response.to.have.status(200);
})
pm.test("Response does not contain errors response", function () {
    pm.expect(pm.response.text()).to.not.include("errors");
});

And the second (errors) test fails as expected:
Sorry… I’m limited in the number of images I can include :=((
image|690x411

I’ve added those same tests to the collection:

I have no variables defined for the collection – they’re all set on the individual queries for the moment. When I run the collection, that query passes:

How I found the problem:
Just running some basic tests to make sure I understood how graphQL testing works in Postman.

I’ve already tried:
Searched for relevant answers here.
Tried changing variable values.

jesii7 · August 16, 2022, 4:42pm

OK; here’s that other image showing the test failure when the individual query is run.

w4dd325 · August 16, 2022, 6:46pm

Hi @jesii7

Im trying to follow your question. I think what you are saying is;

Even though the GraphQL query returns an error, your Postman call is passing with a 200. Is that correct?

If so, I had this same issue recently and after speaking to a developer he told me that GraphQL works differently to REST.

REST will fail with a response code (ie 400, 401, 500 etc), whereas GraphQL passes the information back as part of the query which “technically” is not a failure as it passed a result back to the user. The fact that it passes information about an error does not trigger a failed response code.

If I’m way off base, let me know and I’ll have another read of the above.

jesii7 · August 16, 2022, 8:13pm

Thanks for the reply, @w4dd325 – my question is different than that.

You’re right that graphQL mostly doesn’t return the HTTP status codes that we use for REST; they can return status 200 (when the query doesn’t completely bomb out due to internal errors) and still show errors in the body of the response, as in:

,"errors", {... lots of information here}

Fortunately, Postman provides a way to test the response, which is what I’ve done with this, the second test in my test script:

pm.test("Response does not contain errors response", function () {
    pm.expect(pm.response.text()).to.not.include("errors");
});

In a nutshell, if graphQL gives me a 200 response, but also offers that “errors” element, then that’s an error and the test cause a failure.

It correctly fails when I run the test manually;
It incorrectly passes the test when run as part of the collection.

HTH…jon

michaelderekjones · August 18, 2022, 11:11am

@jesii7 Looking at the following.

Requests and Responses in GraphQL - GraphQL (dgraph.io)

The response should be a JSON file.

It sounds like the errors field only appears if there is an error, so all you really need to do is check for the existence of that field. It looks like “error” will be at the top level in the response so the following should work.

const response = pm.response.json();

pm.test(`Response does not include errors`, () => {
    pm.expect(response).to.be.an('array');
    pm.expect(response).to.not.include('errors');
});

jesii7 · August 18, 2022, 9:08pm

Thanks, Mike… but that doesn’t work. response is not an array; pm.response.text() is also a valid function which returns the text value (essentially JSON.stringified()) which then works with a simple includes test. The test I wrote works (i.e., catches the error) when executed manually against a single query; it fails (i.e., passes the error) when executed as part of a collection.

w4dd325 · August 19, 2022, 4:54pm

I haven’t fed variables into GrahpQL before as you show in your screenshot.

But it could be this that the runner doesn’t like (running out of ideas).

Have you tried adding the variables as collection level variables; So {{variable}} instead of $variable, with values here;

or as a CSV file to feed in for the collection runner?

michaelderekjones · August 23, 2022, 10:13am

@jesii7

According to the API spec, the responses are JSON, so although you could use pm.response.text(), I find it easier to store the response in a Javascript array to manipulate and target elements.

Have you tried the test I posted, as it should be able to sit alongside your other tests\code without affecting them.

Alternatively, can you stick pm.response.text() into the console, and then show what its returning.

You might also want to consider defining the response first, writing it to the console and checking it.

const response = pm.response.text();
// console.info(response);

pm.test("Response does not contain errors response",  () => {
    pm.expect(response).to.not.include("errors");
});

michaelderekjones · August 23, 2022, 10:49am

Looking at the test script examples.

Test script examples | Postman Learning Center

It would appear that the recommendation is to parse the response where possible, but you can use response.text() as an alternative if you can’t parse the data because its not ‘formatted as JSON, XML, HTML, CSV, or any other parsable data format’.

In this situation, you can parse the response body into Javascript, so I would recommend that as the first step.