Collection runner restrictions - Does this include Newman?

This exactly. Sometimes we will end up running a collection a dozen times or more while fine tuning the JavaScript or correcting the import data.

We use runners for more than API testing as well. It works great for support. If we need to fix something on 200 customers, rather than going through and doing it all by hand, we make a CSV and throw it into a runner. And while that saves a lot of time in these support issues, it’s not a feature that’s worth $100/user/month. We’d rather move to a new client altogether. In fact, we’ve already been instructed to start looking into alternatives.

9 Likes

I guess we will find out tomorrow but this will make setting up collection runs for CI jobs incredibly inefficient. We will have to run the job from newman repeatedly until we are sure the collection is stable. And yes, if newman is included in this new limitation we will have to find new tools for API automation.

7 Likes

Has anyone confirmed if the restriction is actually in place yet? I tried running a collection more than 25 times and didn’t stop me.

1 Like

I’m on the paid basic plan and got an email that said it would be enforced on March 15th, the blog post stated Feb 15th. Not sure if thats for the free accounts, or just a typo.

1 Like

Looks like they updated the blog post to clarify. The updated blog post matches what we received in the email. The changes are effective February 15 for free accounts, March 15 for new paid accounts, and your next annual renewal for existing paid accounts. So if you’re lucky enough that your renewal happens before March 15, the new limits won’t kick in for you until 2024.

To better align with the value Postman’s test automation offers for professional testers and quality engineers, we will make the following changes effective February 15, 2023, for free customers, and effective March 15, 2023 for new paid plan customers. Existing paid plan customers will see these limits effective upon plan renewal after March 15, 2023 (except Enterprise users, who have unlimited runs per month):

3 Likes

Just tested it, this policy of limiting the number of collection local runners has taken effect,
detailed policy configurations:

	"collection-runner-usage-limit-scratchpad": {
		"version": 3296,
		"flagVersion": 14,
		"value": {
			"templatePath": "messages/runner/defaults/runner_usage_limit_scratchpad.json",
			"threshold": {
				"notifyLimit": 20,
				"softLimit": 25,
				"hardLimit": 50
			},
			"isEnabled": true
		},
		"variation": 0,
		"trackEvents": true
	},
	"apitest-179-collection-runner-usage-limit": {
		"version": 3296,
		"flagVersion": 3,
		"value": true,
		"variation": 0,
		"trackEvents": true
	},




Good tool, but due to this unfriendly strategy, I think it’s time to leave it.

7 Likes

I’ve missed that thread somehow, to answer the original question: the limit doesn’t apply to Newman, but only to manual runs.

I’ve forwarded the rest of the feedback internally.

2 Likes

This is an insane change.
We use the Local Collection Runner to automate our login flow (6 requests) and this change means we’ll have to click through them manually.
This will force us to move to another service. It’s a shame, really, since I really enjoyed using Postman…

6 Likes

@joint-operations-obs I’d be interested in hearing what your auth flow looks like, there are a few options for you to reduce the amount of requests it requires or to use it externally, e.g.:

  • Have you checked that your auth flow isn’t supported by the authorisation helpers built-in Postman?
  • Run the flow in Newman/Postman CLI if it is to retrieve a token that needs to be subsequently used. You can even pair it with the Postman API to automatically update your environment variable.
  • You can otherwise run your requests as part of the pre-request scripts and put them at the collection-level, this means the auth flow will be executed before every request in the collection, and you won’t need to run manually every time

Hi,
Our APIs use HATEOAS architecture. We map the response from “root” API links to environment variables which are used in the login flow.
Out of all the suggestions you gave, only the pre-request script setup would work for us. But that’s ultimately a hack.
Might as well write a bash script that does the login flow and copy access tokens to Postman env. But what’s the point of paying $15 per team member if we need to resort to hacks?
Our team is small, and there’s free options that are capable of chaining requests without the 25 run limit, therefore goodbye.

1 Like

@joint-operations-obs Please reach out to the support team (help@postman.com) and let them know of this feedback, this will be a better channel for it as the community forum will only be able to give you workarounds.

@arlem Thankful that newman runs will remain intact but this is still very troublesome. We are basically left with testing our full collection runs by checking them into source and running them through newman to iron out any issues with our tests. This will add a lot of overhead to creating clean test collections and will dirty up our CI job with failures that should be caught before checking into source. It really is an unfriendly way to push us into licensed Postman clients.

5 Likes

@arlemi Thank you for your forwarding.
Can you give feedback to the community on how Postman will handle these situations?
Will this limitation still be retained?

3 Likes

I assume (hope) this decision will be reversed, or Postman risk badly hurting their reputation.

6 Likes

It has caught us completely by surprise in the middle of a critical testing process. If this is not reversed we will move on to another tool (like SoapUI). We are not going to give second chances to find in the future that they limit later Newman or CLI if they dont get enough pay users.

What an error, Postman Team

7 Likes

Huh? I didn’t get the news until the paywall hit me. I’ll give them a month to revert this changes before i switch to an alternative.
Though I don’t want to start from scratch again, i don’t want to stick to a tool that feels like ransomware either… :confused:
And who knows what their next grand idea will be.

4 Likes

Making us pay to use the button in the UI versus running Newman in a shell is the worst product strategy move I have seen in a long time. Should we learn to use curl directly? It’s time to go test the competition The Collaborative API Development Platform - Insomnia

2 Likes

Wow, it has individual plan.

@yann Wow, it has individual plan.

No more tricks, true solution is here.

1 Like