Checkpoint testing in PM

Help Hub
1

How are response values checked in PM?’

We can check the response structure, properties, headers and things like that.

How do we validate that the data IN the response is correct?

For example, how do I test that the number of customer objects in our response is correct?

Say we have 5 customers in the users db. I want to run a Get /customers and check to see that the response did in fact have 5 objects in the response array? Hard code what we expect the value to be?

Psuedo code
Get /Customers //returns an array of all the customers in the database.
//Retrieve the actual number of customers from the db (essentially select Count(*) from customers) and store it in a variable
//compare the number of items in the array to the value in the variable.

This doesn’t seem to be something native to PM.

Is this type of testing even correct for API testing? If not, what should we be doing?

2

What have you looked at doing or what do you have in place?

In the most basic form of checking a value in a Postman test, you would need to do this:

pm.test("Your test name", function () {
    pm.expect(pm.response.json().someProperty).to.equal(100);
});

I expect this property to equal this value

I reference @vdespa tutorial quite a lot because it’s a great starting point to get starting with creating test in Postman.

1 Like
3

Yeah, I’ve watched a lot of @vdespa videos. They are awesome!

Your example is just hard coding an expected value. Which isn’t really dynamic or easily maintainable.

FYI We do have some tests exactly like the video shows. Simply checking a property value is pretty straightforward.

What if I add a user? I’d have to find that test and add 1 to it? Boo! :slight_smile:

I understand that in “real life” the client calling the api wouldn’t know about the DB.

For example, the postman api. We can run requests against it, but I can’t directly view or connect to the database.

But this is needed to test if the api is retrieving the correct item\s at all.

I’ve seen this in desktop testing platforms like Test Complete. You have the ability to create “checkpoints” to test the expected value with the actual value.

Just query the db and then put the value in a variable and check against it during the test run.

Does PM have anything like this? Do we need a 3rd party utility for this?

4

These are not tests, these are checks.

You’re asserting against something that you know the answer too. If something were to happen that impacts the known response of the endpoint, these are like miny change detectors that will tell you.

Your test needs to be deterministic, in some way, in order for you to assert against something known. If things are highly dynamic and changing all the time, it’s going to be hard to maintain but you’re also going to lose the focus of what the test is testing.

If you need something more dynamic you could look at doing data-driven testing, where the test data is creating these different things that you’re looking to assert against.

I have 10 items in my data file, I use that data on the create endpoint, when I call the users endpoint, I expect there to be 10 users created.

Ultimately, what are you testing for? What is the risk that you’re trying to mitigate against?

I would take a step back from the tool and work out what are the important critical areas of your API and design your tests around those.

5

Ok, let’s do it.

Yes, checks (aka checkpoints) are one aspect of testing we need.

Ultimately, what are you testing for?
We are testing that the data it’s returning is correct.
If we have 5 customers in the db and we hit the endpoint Get all_customers (fake endpoint name) and the response is only 4 customers, we know that something is wrong.
If the endpoint (thereby the db it’s connected to) is not returning the same number, we’ll know there’s a bug in our endpoint.

What is the risk that you’re trying to mitigate against?
The risk we’re trying to mitigate is that the endpoint is not returning what it should. I can test a response for the correct structure, header values and that the json has the correct data types where we expect them. “Who put an string in my bool?”

We can do this pretty easy because our endpoints structure isn’t changing that much. We also have our API defined in the API section, so it’s going to point out a lot of those issues.

In addition to the response itself, we need to validate what’s IN the response against the database it came from to ensure our endpoints are returning the correct values

If we only used one set of test data for all time and didn’t change it, this wouldn’t be a problem, but say we have an endpoint called Post new_customer that adds a customer to the db.

This test would fail after we added the new customer because it expects the count to be 5 and it returned 6.

pm.test("Your test name", function () {
  //Psuedocode
    pm.expect(//array of customers.itemCount.of(5)) not sure how we'd do this.
});

I guess the easiest way to explain this is we need to test what the api returns in terms of the correct keys AND the correct values that come along with them. If our data never changed, this would be easy because we could hard code it like you suggest, but that’s not the case. We may need to attach or API to a specific customer (of ours) and use their data. If we hard code these values, all the tests would break and we’d have to hard code those values.

Does that make sense?

6

You don’t or shouldn’t. You have other testing layers responsible for ensuring this (probably some functional tests). API testing / Postman should not care or know what is happening behind the scenes.

You could add one customer and check the behaviour (there list contains one additional customer). Or delete it, … Try to test the behaviour of the API, not some assumptions about the state of the database.

Use Postman to test the structure of the response (with JSON schema) and other aspects relevant for your API clients.

When you later discover bugs, think which is the responsibility of each testing layer before writing tests.

1 Like
7

Your example would work if I was testing to see that a post request worked and added a customer.

What I’m trying to say is, how do I know that Get /customers is returning the correct number of customers in the first place?

I need to validate what the api is returning against what the actual value is.

For example, I want to check the total invoice balance for invoice id 42.

So I do a Get /Invoice_balance/42

This call returns an invoice balance of 100.00.

How do I know that the value 100.00 is correct without looking in the db?

I’d have to manually do a
SELECT InvoiceBalance FROM Invoices where invoiceID = 42
in our DB ide to verify that the 100.00 is correct.

What if the actual value from the query above is 200.00?

My api tests could all come back as PASS, but the balance is totally wrong.

How do we do this kind of test if PM doesn’t know about the data it’s supposed to be returning?

Other platforms address this issue with database access which allows a query return to be stored as a variable to test against.

Is there a way to somewhat replicate this in PM?

8

You need to clarify which testing levels you have and what is their respective responsibility.

It is not the job of API testing to check such aspects. Write unit tests and functional tests to cover such aspects.

I consider it to be an antipattern check-in with the database behind the scenes from your API tests.

9

Ok. How would you suggest we do this?

How do we ensure the api is returning correct values without testing the api response against what’s in the database?

10

Hi @robmcec

I think you bring up great talking points that many struggle with testing modern day API implementations.

As @vdespa has suggested, I agree it’s more of an anti pattern to directly query the database to validate values.

In my personal opinion, and based on my experience, testing is done best when starting off with a clean slate, or a clean instance of the environment, database, etc. It becomes very difficult to truly validate the backends are properly working when the values exist in there without the scope or knowledge of the front end client (in this case Postman), and then a teardown when you are finished, restoring back to its original clean state.

I have always found it easier to initialize my test data using the client that will do the validation. Such that, referencing your prior example, to test the creation of a user, I should first create that user using the client (Postman / your API) and then validate the user was indeed created via a second call, most likely a GET. Same thing with the invoice; it should first be created via the client, and then tested against in a second call or a GET. In your initialization of those values, you can dynamically store the values that you create, so that when you test to see if it actually was created, or the GET, you can validate against those values.

This way its soup to nuts, and your client is agnostic of anything that has happened prior, it just knows what it created, and what it received.

I believe this truthfully tests out the system well enough to validate its behavior.

Now if you don’t actually have an endpoint to validate what has been created via the client, then that becomes a bit more difficult to actually validate, because your testing mechanism will have to be outside of the scope of which the data was initialized to begin with.

I hope this makes sense, or that I wasn’t rehashing something earlier.

Best,
Orest

2 Likes
11

From my experience, I did not encounter issues at this db level without being detected at some state of testing, without necessarily writing an explicit test for counting items in the database.

Move this kind of testing to other testing layers. They may do a better job.

Also, test based on risk and experience (aka discovered bugs). You will rarely have resources to test everything that may go wrong.

1 Like
12

Thank you so much for your thoughts on the matter!

I guess we have some thinking to do when it comes to api testing.

May I ask what your opinions are on api endpoints (non-customer facing) specifically for testing?

These endpoints wouldn’t have business logic in them, they’d essentially be a query that returned db records.

For example, we’d have an api call that returned all invoices joined to the table that contains there invoice lines. That way, we could parse the response and it would essentially act as if we were accessing the db. The key here is no business logic, just data to compare our customer facing endpoints with the db values.

13

My word of advice on that - don’t build anything that would devastate you if it was compromised. Having an entry point to ALL data like that (even hidden/undocumented) is just asking for trouble.

I was just reading through this post, and the standard way to do this workflow - checking if the APIs are returning the correct values - is to seed that data from within your workflow.

Example:

  • Request 1 - Generate test values and save them to variables
  • Request 2 - Use the variables to add a new record in the system. Save the id of the new record as a variable
  • Request 3 - Use the id variable in a GET to load the details of the new record, and make sure they match the variable values from request 1

This way, you don’t care about any data that is already in the system - which really is how it should be. You’re testing that the current implementation is correct.

1 Like
14

Thanks Allen! I ended up doing this last night.

I arranged my api calls so the ones that don’t have a unique identifier get called first.
I retrieve the first ID from the return of that call and set it as an environment variable.
I then use that environment variable for all subsequent calls that require that id and then test the response for that id. Although, it would probably fail if the Id wasn’t correct anyway.
When the “workflow tests” are complete, I unset the variable from the environment.

We don’t have a lot of POSTs right now (it’s a very new api), but I’d probably put the POST prior to those tests and then use the id in the subsequent calls.

Does this sound correct?

15

Yup, that’s right! Nice work!

16

Woohoo!

Thanks to everyone for their help with this

1 Like