Collection runner restrictions - Does this include Newman?

Reguarding this article Announcing Postman 2023 product and versioning updates…and VS Code extension! | Postman Blog

As of Feb 15th:

  • Local Collection Runner: Runs will be available with Free and Basic plans at 25 runs per month, Professional plan at 250 runs per month, and Enterprise plan at unlimited runs per month

Am I reading this right? I’ll only be able to use the collection runner on my desktop application 25 times a month? Will this be enforced with Newman as well?

10 Likes

That’s how I read the post too, and they even confirmed it with an email sent to our team. It’s an awful change, very hostile and anti-user. There’s absolutely no reason that calling local APIs with a local runner should cost anything. Just another attempt to drag every penny they can out of users.

15 Likes

Newman has been more than a year since its last release, at the same time, the Postman CLI was released and received continuous updates, I don’t know if Postman will end support for Newman in the future and force users to bind to Postman CLI, because the Postman CLI can not run independently, it must require a login and consume Postman API counts at the same time.
If that is the case, then that’s really bad news for a free/basic account.

4 Likes

This will force our team to move away from Postman Cloud.

Please reconsider this. Postman does not have the competitive edge in the market to make this change, it’s going to greatly hinder the developer and QA experience.

There are plenty of options and let’s not ruin a great product!

I’m not sure how this can be put into place with current annual plan holders. This was a free feature, that is now behind a paywall.

12 Likes

Yes this is unacceptable, just got the email as well…we subscribed to the Basic (PAID!) plan for one year in January, then they make this change out of the blue a few weeks later…
This is very unfair, we would have never subscribed if we had known this, as it makes the basic plan unuseable.
If it affected only the free plan, sure, they can do whatever they want with that one (still a bad descision imho), but for existing, paying basic users? This is not what we subscribed for.
I really hope they reconsider…
I tried to ask them about this on the blog comments, whether it also affects existing subscriptions, and never got an approval for my question. :face_with_raised_eyebrow:

8 Likes

What is meant by “Local Collection Runner” in this context?

Does it mean “anything run in the collection runner”. The manual runs that use zero Postman Cloud resources?

image

Surely not, as it would make the tool nearly useless for free\basic users.

Some functionality like setNextRequest and data driven tests, only work through the collection runner.

I’m assuming that it won’t affect tests run from the Newman command line, but you’ll burn up over 25 requests testing your collection before setting up up in Newman.

Am I missing something here?

8 Likes

From their official email:

“What qualifies as a local collection run? Anytime you choose “Run manually” in the Collection Runner, it’s considered local.”
They even provided a similar screenshot as you in the email.

Unfortunately you are not missing anything. And yes, it makes the basic plan effectively useless.

8 Likes

Oh dear, that sounds Nuclear. You are literally running it locally, and not consuming any Postman Cloud resources. That is crippling the product to push paid for versions.

The collection is the Test Suite, so you are literally saying, you can’t run the whole test suite unless you have a paid for version.

Even the professional version with 250 requests per months wouldn’t be good enough.

Oh well, it was good while it lasted. :slight_smile:

8 Likes

The only clarification is I believe that means 25 complete runs of your collection, not 25 requests in a collection. What I still need to know is if this will be enforced on Newman, if you have your collection set up to run daily via cron or some CI tool this is a big problem.

8 Likes

not to mention that during the development of scripts for a new test case you might want to run parts of your collection to check the script a few times.
And as far as I can see it, even if you just run one folder inside your collection it counts as a collection run.

9 Likes

This exactly. Sometimes we will end up running a collection a dozen times or more while fine tuning the JavaScript or correcting the import data.

We use runners for more than API testing as well. It works great for support. If we need to fix something on 200 customers, rather than going through and doing it all by hand, we make a CSV and throw it into a runner. And while that saves a lot of time in these support issues, it’s not a feature that’s worth $100/user/month. We’d rather move to a new client altogether. In fact, we’ve already been instructed to start looking into alternatives.

9 Likes

I guess we will find out tomorrow but this will make setting up collection runs for CI jobs incredibly inefficient. We will have to run the job from newman repeatedly until we are sure the collection is stable. And yes, if newman is included in this new limitation we will have to find new tools for API automation.

7 Likes

Has anyone confirmed if the restriction is actually in place yet? I tried running a collection more than 25 times and didn’t stop me.

1 Like

I’m on the paid basic plan and got an email that said it would be enforced on March 15th, the blog post stated Feb 15th. Not sure if thats for the free accounts, or just a typo.

1 Like

Looks like they updated the blog post to clarify. The updated blog post matches what we received in the email. The changes are effective February 15 for free accounts, March 15 for new paid accounts, and your next annual renewal for existing paid accounts. So if you’re lucky enough that your renewal happens before March 15, the new limits won’t kick in for you until 2024.

To better align with the value Postman’s test automation offers for professional testers and quality engineers, we will make the following changes effective February 15, 2023, for free customers, and effective March 15, 2023 for new paid plan customers. Existing paid plan customers will see these limits effective upon plan renewal after March 15, 2023 (except Enterprise users, who have unlimited runs per month):

3 Likes

Just tested it, this policy of limiting the number of collection local runners has taken effect,
detailed policy configurations:

	"collection-runner-usage-limit-scratchpad": {
		"version": 3296,
		"flagVersion": 14,
		"value": {
			"templatePath": "messages/runner/defaults/runner_usage_limit_scratchpad.json",
			"threshold": {
				"notifyLimit": 20,
				"softLimit": 25,
				"hardLimit": 50
			},
			"isEnabled": true
		},
		"variation": 0,
		"trackEvents": true
	},
	"apitest-179-collection-runner-usage-limit": {
		"version": 3296,
		"flagVersion": 3,
		"value": true,
		"variation": 0,
		"trackEvents": true
	},




Good tool, but due to this unfriendly strategy, I think it’s time to leave it.

7 Likes

I’ve missed that thread somehow, to answer the original question: the limit doesn’t apply to Newman, but only to manual runs.

I’ve forwarded the rest of the feedback internally.

2 Likes

This is an insane change.
We use the Local Collection Runner to automate our login flow (6 requests) and this change means we’ll have to click through them manually.
This will force us to move to another service. It’s a shame, really, since I really enjoyed using Postman…

6 Likes

@joint-operations-obs I’d be interested in hearing what your auth flow looks like, there are a few options for you to reduce the amount of requests it requires or to use it externally, e.g.:

  • Have you checked that your auth flow isn’t supported by the authorisation helpers built-in Postman?
  • Run the flow in Newman/Postman CLI if it is to retrieve a token that needs to be subsequently used. You can even pair it with the Postman API to automatically update your environment variable.
  • You can otherwise run your requests as part of the pre-request scripts and put them at the collection-level, this means the auth flow will be executed before every request in the collection, and you won’t need to run manually every time

Hi,
Our APIs use HATEOAS architecture. We map the response from “root” API links to environment variables which are used in the login flow.
Out of all the suggestions you gave, only the pre-request script setup would work for us. But that’s ultimately a hack.
Might as well write a bash script that does the login flow and copy access tokens to Postman env. But what’s the point of paying $15 per team member if we need to resort to hacks?
Our team is small, and there’s free options that are capable of chaining requests without the 25 run limit, therefore goodbye.

1 Like