Which data does Postbot/AI access and is it potentially shared to others?

đź—Ł General Discussion
1

Hi,
I have tried to find documentation on Postbot/AI which can be used to write test scripts. The main focus is exactly which set of data is used by the AI, and equally important, if the data leaves the local machine and is potentially ingested into a larger dataset out of our control, or the AI works in a sandbox manor locally.

Can anyone provide insight into this area, and perhaps point me in the direction of descriptions etc.?

It looks like only the last response body and likely headers are made available to the AI, but I would like to get this confirmed.

That said, the tool looks pretty cool, and works quiet well as an easy to use way of introducing test scripts. This is especially true if the technical knowledge does not allow for an employee to write it from scratch, which can take time and a more technical knowledge than some testers have in the backpack.

Regards
Søren

2

I can’t answer the question about if where your data goes, whether it stays local or not, but in my opinion these types of AI tools for generating code should be used for testers that know the underlying basics. The tools save the tester time, but the tester still needs to double check each of the tests for accuracy and there is usually a level of refactoring required. Therefore the tester needs to know how the tests work manually first.

They shouldn’t really be used by testers who have no experience who won’t necessarily understand the code that is written. Get those testers to learn how to do things manually first and then introduce them to the AI which will help write test suites quicker and more efficiently.

You are basically asking for technically unqualified staff to do something that is currently above their skill set which is a dangerous precedence to set (again, this is my opinion).

Use the AI as a learning tool by all means but don’t use it in production like this unless you have a strong review process. Concentrate on the basics first with your testers.

Learn to be come a good tester first (the Why). ISTQB courses, etc. Then learn the how. HTML, CSS, JavaScript, REST\JSON, Soap\XML, Then learn the tools, Postman, Cypress, Selenium, etc. Have a roadmap and plan for learning these skills within your org, and start your testers on that path. AI should be at the bottom of this list after you have learn the why and how.

AI will absolutely change the way that development is done. But use it as the tool it is, which isn’t to replace a skills gap (again, in my opinion).

3

Hey Søren, this is a great question.

Postbot only sends the shape of the JSON from the response and existing scripts (if any have been written), all other data is stripped prior to being sent. For instance:

{
  body : {
    'first_name': str,
    'last_name': str,
    'age': int,
    'siblings': [],
    'pets': {
      'cat': {
        'name': str,
        'breed': str
      }
    }
  }

The only caveat is if a user provides specific information in the prompt to Postbot. For instance, “write a script to check if the returned value of first_name is Sterling”, in which case Postbot will use this information to create a test script.

In regard to whether Postbot data is used to train other AI models, we have agreements with our AI partners in place that prevent them from using any data to train new or existing models, nor do they store any data that we send across the wire.

I hope this helps!

Sterling
Engineering Manager at Postman

3 Likes
4

Is there any official documentation about this?
One or more links would be awesome.