250 collection runs per month on Professional?

kyle-revio · 21 February 2023 06:18

We use postman to manage and run automated end to end API tests. Can we achieve this with Insomnia?

norrispostman · 21 February 2023 07:59

Insomnia is the most similar tool on the market, but this ought to be a lesson to people to not put all their eggs into a commercial offering that can have functionality pulled at a moment’s notice. We’ll be moving to a code-first solution rather than continuing to put ourselves at the mercy of Postman, or trusting Insomnia not to pull a similar stunt.

itneverhurts · 21 February 2023 09:51

No. Forget insomnia, it’s very convoluted. The free version is fine for a collection of calls, but there’s not scripting or verification or anything built in. And you can’t even log in and sync without being on a paid plan. This one died before it was really born.

ign-cloudferro · 21 February 2023 10:19

Is there any possibility that 25 collection runs limit will be withdrawn?
We have over 500 tests in collection that heavily rely on setNextRequest and this change will kill us. We use local runner for debugging and for developing new tests. I personally use about 100 local runs per day. Please share any information regarding status of that decision. Should we really give up on Postman as a tool?

spacecraft-partici10 · 21 February 2023 10:37

I’ve not checked it out properly, but if you’re on Mac, someone at my company suggested RapidAPI for Mac instead.

norrispostman · 21 February 2023 10:57

The really funny thing about this is that support have confirmed to me that the 250 runs on professional is counted for the entire team not per user.

We’d actually be better off going back to the free plan so we get 25 runs per user.

The message from Postman is very clear: Upgrade to Enterprise or go away. We’ll do the latter.

kyle-revio · 21 February 2023 12:48

@arlemi are Postman CLI runs affected by this limit?

arlemi · 21 February 2023 13:56

@kyle-revio Postman CLI runs aren’t affected by this limit. They’ll count against the “calls to the Postman API” quota (see pricing page).

@norrispostman As far as I’m aware the limit on the free plan is also per team, as long as you’re in a team.

@ign-cloudferro Have you looked at running your collection as a scheduled run or using the Postman CLI or Newman? Sounds like you may benefit from it instead of running a collection that big manually every time.

supply-administrato3 · 21 February 2023 14:13

Hi,

I am Product Manager with Postman and I was going through this thread.
@norrispostman Please send us more details at help@postman.com so we can understand your situation better and support you. Our goal is to enable your workflows as seamlessly as possible while you get time to respond to the change. If you wish to discuss this instead, please use this calendly link to schedule some time with me. We would like to understand your current workflows and how you use Runner.

The changes will be effective on March 15 if you are on our basic/professional plan and on Feb 15th if you are on the free plan (the limit will be effective post your plan renewal happens i.e. existing plans or plans purchased before March 15th will have unlimited runs until their renewal date)

@jon.krantz Please send us a mail at help@postman.com so we can help you get unblocked for your release and while you get time to respond to the changes.

supply-administrato3 · 21 February 2023 14:17

Hi @ign-cloudferro Please use this calendly link to schedule some time with me. We would want to talk to you and understand how we can help you. I understand that 25 runs limit might be affecting your workflows suddenly and we want to make sure we unblock you so you can understand these changes better.

vdespa · 21 February 2023 15:24

I feel that this new limit makes little sense and only hurts this amazing Postman community. It can also deter others from adopting Postman in the first place.

Anyone using Postman with the intention of automating runs knows that using the Collection runner is the first tool that is being used to test if an automated collection run is possible (setNextRequest, external data files, variables, etc.).

It is also the most helpful tool for debugging a collection that fails to work. Since it is integrated into Postman, it makes inspecting the requests/responses/console much easier compared to Newman or any other tools. It is an important productivity tool.

Anyone who has used the Collection Runner to prepare a collection for automatic runs can easily understand that 25 runs can be consumed even in an hour of testing. So the limit of 25 collection runs essentially kills the usefulness of this tool. What can be more frustrating that having to stop work because you have hit an artificial limit? Even 250 runs on the professional plan is very very little.

To properly do automation, the Collection Runner is key. For a team, hundreds, if not thousands, of runs are needed. How can people see the value in paying for more advanced features if they can’t even get something simple to run? If somebody can’t automate a collection run locally, I doubt they will spend the time to use a CLI tool.

This limit of 25 collection runs is completely misaligned with other limits that the free tier has, like 1000 API calls.

Since this feature does not seem to use any Postman cloud resources, it also seems unfair. What is the justification for this change? The jump from unlimited usage to 25 runs is way too sudden and just hurts Postman as a test automation tool.

I understand that Postman needs to earn money to keep the product and the development efforts going. Postman has grown a lot over the years. We all want a financially stable company to take care of the product.

At the same time, the product’s users need to see the value in paying for the product. From this thread, it seems that many already paying customers no longer see the value, which is very sad.

ModoJohnny · 21 February 2023 16:14

@supply-administrato3 When you say “existing plans or plans purchased before March 15th will have unlimited runs until their renewal date”…is this “renewal date” when we play the next monthly invoice (we are on the monthly basic plan)? Which isn’t really a renewal, just paying a bill. So until we cancel or change plans, will we continue to have unlimited Collection runner runs?

ray_x · 21 February 2023 16:29

Completely agree with you, this unfair change is indeed disgusting to users, it takes away its basic productivity, infact before this change I was going to upgrade to a paid plan, but now it seems unnecessary, perhaps that is the value of who made the limit, I’m also sure @supply-administrato3 won’t say here the justification in which made this change, right?
Only the valueeeeeeeeeeeeeeee…

kyle-revio · 21 February 2023 17:21

Correct, i am in the same boat. I purchased an annual Professional plan which expires in August.

jason-omx · 21 February 2023 23:51

I assume this will be walked back a good bit or Postman risk seriously hurting their (good) reputation.

kyle-revio · 22 February 2023 06:46

We all hope so, but as engineers we have to work with worst case scenario. So i have until August to find alternatives and migrate.

preetham.m · 22 February 2023 19:02

Thank you all for your feedback. A majority of our customers chose the Enterprise plan to run and deploy quality and security gates through a combination of manual and automated tests. We set plan thresholds in a way so customers only upgrade when they find value in the product. We understand that some of our customers might have use cases that cross those thresholds but also don’t find enough value to upgrade. Please get in touch ,at help@postman.com, with us so we can understand your use case better

ray_x · 23 February 2023 02:09

“We understand that some of our customers might have use cases that cross those thresholds but also don’t find enough value to upgrade”.
Well, let me translate it, “All you guys should upgrade to the not enough value enterprise plan”, that’s the value.

kyle-revio · 23 February 2023 07:46

The use case is literally to just run things locally.
Your limitation on local runs makes no sense. Any other limitation would make sense, like Postman API calls or Storage usage.
This is a bullshit response and you know it.

ray_x · 23 February 2023 08:23

@kyle-revio I found something interesting, Postman is collecting data on the runner’s runs, via

{
	"service": "usage",
	"method": "get",
	"path": "/usage/operations/collection_run_limit"
}

{
	"operation": {
		"allowed": false,
		"limit": 25,
		"remaining": 0,
		"usage": 25,
		"overage": 0,
		"spillage": 99999999,
		"allowOverage": false
	}
}

It is not known when this collector was added,
therefore, @supply-administrato3 @preetham.m it is not possible for Postman to be unaware of the user case, instead it’s possible that Postman has collected a lot of this operational data to make this limit for users.

Another question is why still send tracking data to these servers even if this option “Send anonymous usage data to Postman” was turned off?
for these trackers?

analytics.getpostman.com
bam.nr-data.net
insights-collector.newrelic.com
js-agent.newrelic.com
o1224273.ingest.sentry.io

After doing some research, I was also concerned about what it says about data security.