250 collection runs per month on Professional?

michaelderekjones · February 23, 2023, 10:33am

What is emailing help@postman.com going to achieve?

You know what the use case is for the runner. It’s your product, you designed it.

If you are in doubt. Your product has some key elements that only work in the runner. The ones that are useful for automated testing, including setNextRequest and data driven tests.

This runs locally using the Postman desktop client, and until recently had no limits.

You’ve imposed a hard to understand limit, which makes the feature nearly unusable for everyone outside of an Enterprise plan. Even the professional plan limits look artificially low as its based on the team.

For pretty much everyone here, it would appear that you have done this solely to push the more expensive Enterprise plans. If you haven’t, then it would be nice to hear the rationale behind this move.

As mentioned previously, if a company is willing to pull or severely limit a feature that has been free for over a decade, then what other feature may you restrict in the future? (For example, pulling the support for Newman over the CLI, or perhaps sendRequest() next.

sendRequest uses no local resources, just like the Collection Runner, so I’m guessing its fair game.

The main issue here is a loss of trust.

I’ve only been using the product for six months, and was starting to get colleagues within my organisation to invest time to see if we should adopt more globally. We were using the free version and sticking the code in our code repository which was all we needed for our current requirements as we don’t design many API’s, we just consume others so I don’t need the documentation features.

There is always a risk in using free software, that it will be pulled or deprecated and that is always the risk you take.

However, now knowing that Postman appear to be in this camp. I cannot realistically continue to champion the product internally as I no longer know if key features may be removed or limited in the future.

This probably means we will now look for paid for software, but I doubt I would recommend Postman as an option because of what has happened here.

Enterprise is too expensive for our use case. Making you more expensive than SoapUI.

Adding new features to the top tier like your new CLI to streamline integration. That would be fine. But removing or severely limiting existing features is not a business practice that I can recommend.

docking-module-part1 · February 23, 2023, 1:33pm

Interesting…maybe we could figure out how to block that traffic from going out so that we dont hit the limit?

ray_x · February 23, 2023, 2:07pm

@docking-module-part1 We can indeed do this with some hacking tricks, but hopefully Postman can design reasonable and competitive subscription plans instead of playing tricks.

@supply-administrato3 the only product manager on the community, right?

supply-explorer-3481 · February 23, 2023, 2:10pm

@preetham.m This is an infuriating response. You have a lot of people right here trying to communicate their use case right here, can you please respond transparently?

Can we have a direct response as to the justification why there is a limit on using local resources?

Every day that goes by with more doublespeak erodes the trust that you have built up over the years. Postman used to be a no-brainer, but this boneheaded move is really transparently greedy and disappointing.

pvinten · February 23, 2023, 2:51pm

Respectfully, this smacks of driving users to your expensive enterprise plan.

I can understand why you would restrict runs on a free tier, you are running a business after all , but given that this is a restriction includes localhost runs, and runs on restricted environments, it feels stingy.

250 runs for a £36 per user per month is very steep. I only use Postman for QA of my company’s APIs. I could easily hit 1000 runs a month when I’m busy, probably more than that. My work covers exploratory testing, field assertions and end to end regression testing. It’s 3600/250 = 14.4p per run. If I want to generate Bearer Tokens, that’s 14p. If I want to validate a change, using a regression pack, that’s another 14p.

I pay for pro tier as I understand the need to support the developers of the tools that I use. I also understand why you would have a higher level of cost for users hitting 100k runs per month, but I’m going to have to look elsewhere. It’s a shame but I just can’t justify the cost if the restriction is to remain that severe.

ray_x · February 23, 2023, 4:08pm

You’ve copied the same words on GitHub.
Let’s keep track of this issue.

github.com/postmanlabs/postman-app-support

Remove collection run limit for non scheduled cloud based runs.

opened 02:14PM - 20 Feb 23 UTC

byrdman

feature collection-runner

### Is there an existing request for this feature? - [X] I have searched the ex…isting issues for this feature request and I know that duplicates will be closed ### Is your feature request related to a problem? My development cycle has me running collections from within Postman multiple times during the day, I can quickly reach the new 25 max run limit in a morning. I don't understand why I'm limited to the # of collection runs I can do locally but not if I use the cloud runner. My API isn't accessable to the cloud runner. This has ruined a great tool. The only way I can continue to use Postman is to either upgrade to the Pro license or do all my runs using the CLI (which removes many of the benefits of using Postman). ### Describe the solution you'd like Remove the limit for non-scheduled cloud runs. It doesn't cost you any resources for us to run collections 1000 times on our local hardware (which you now limit to 25 times per month) vs doing scheduled runs from your cloud (which doesn't work in our environment). ### Describe alternatives you've considered MIgrating off of Postman Writing my own tool based on the CLI. ### Additional context All of the "push everything to the cloud" has ruined the great product that Postman once was. It took over a year to get the performance problem of importing large collections fixed since it was uploading it to your cloud. There was no reason for it to be uploaded to the cloud since Postman doesn't play nice with our git system (it's not publicly accessible therefore Postman can't talk to it so there's no reason to collaborate in the cloud).

andrew.keynes · February 24, 2023, 2:36am

Just encountered this myself - these limits seem crazily low and pretty much make the product unusable for me.

If someone had typoed ‘Month’ instead of ‘Day’ I could understand it, but even then 250 runs in a day would be quite limiting if you were developing a collection of tests.

As the counter increases every time you hit the button, if you’re writing a new scenario with some pre-request scripts that need debugging you could run a whole bunch of times just to figure out what your bug is.

As a free user currently I could potentially be swayed to paying for the pro plan (If it was per-day) but the enterprise plan is completely out of my price range.

Patrick.H · February 24, 2023, 11:28am

For me postman was a very important tool the last 2 years and even considering that i’ll have to rebuild everything with another tool pains me greatly.

The paywall to an existing feature was very sudden and it hit us in a critical stage of development. Therefore i can’t help but see some resemblance to ransomware. Luckily i’m used to Newman and I can make do with the Newman workaround for now. But what does the future hold?

I don’t want to rely on the whims of the creators of the tool that i’m using. Abandoning postman will not be without difficulties, but if Postman decides to lock existing features on such a short notice with so little reasoning and transparency, the transition to another tool will become a necessity.

spaceflight-candidat · February 24, 2023, 10:06pm

I have to reluctantly agree with Valentine - in my personal network I have never seen a Postman update land so badly. Reading a lot of blogs, comments and general posts about this. None are good.

Postman is a tool that I have a long relationship with and am a huge fan of. The community and positive regard for this amazing platform is unparalleled.

People feel very unsupported by this change, and I’m gutted for the team at Postman who have to support it. I hope all the bad press and feedback will encourage a rethink.

vesu · February 26, 2023, 8:28pm

I’m following on a brilliant course about Postman and API testing by vdespa. I was experimenting with collection runs and from my experience 25 runs are not enough even for learning purpose.

I was evaluating Postman for API test automation in my company but I doubt that Enterprise plan will be within the budget. Basic/professional plans are more affordable for small companies but with these limitations they are not useful at all.

yann · February 27, 2023, 10:47pm

This is a sad moment for the Postman community. How is this pricing move aligned with Our Open Source API Philosophy | Postman Apart from the button in the UI how much of the local run is dependent on the underlying open source projects?
Work on load-testing features with cloud agents and make people pay monthly for actual cloud resources.

docking-module-part1 · February 28, 2023, 6:36am

This is precisely the point. I wish Postman could comment honestly on it? @arlem @supply-administrato3 @preetham.m

ray_x · February 28, 2023, 1:00pm

Well, I’m already pretty sure this limit was enforced after Postman 10.10.x.
So far, there is still no official explanation for users in this community.
I also don’t want to discuss the rationality of this disgusting limit.
Not only that, there is also no buffer time for free(even if basic&pro) users to accept this change, especially users who depend on Local Runner to debug test script.

If you guys want to avoid this limit,
the urgent workaround is to downgrade to Postman 10.9.4(This shoud be latest version that is not affected by Runner limit, I have tested it).

So let’s do this,

1. First, Backup/Export your Collections/APIs, if you sycned data to cloud, you can skip this step.
2. Block Postman auto upgrade by adding host 127.0.0.1 dl.pstmn.io in C:\Windows\system32\drivers\etc\hosts
3. Uninstall Postman v10.10.x and Install Postman v10.9.4, I have uploaded to Cloud Storage (to avoid Postman potentially maliciously deleting this old version)
4. Start Postman and import your data or login your account.

Download

You will see there is no limit any more for Local Runner.
Note:
This is only urgent/temporary solution.

Patrick.H · February 28, 2023, 1:38pm

I’ll stick to Newman for now but i’ve saved your Instructions for later use since i’m pretty sure your post is going to be deleted.

Thanks for the effort!

ray_x · February 28, 2023, 1:57pm

Seriously? I don’t think they’re going to pay attention to this just as they don’t care about users, let’s see it.

alekdavis · March 1, 2023, 4:08am

From https://www.postman.com/company/open-philosophy/:

Postman has always believed in open dialogue with our community; we continuously make improvements and add new features based on feedback from our community of more than 25 million developers. We are transparent with our roadmaps, we constantly communicate our issues, and we support all major API specifications. That’s the Postman open philosophy. But it’s more than just a philosophy. We put our commitment to openness into practice every day. Below, you’ll find details on a selection of these real-world efforts and projects.

I wonder which community feedback asked for the 25/250 monthly run limits.

arlem · March 1, 2023, 9:33am

@ray_x We do read all the feedback that’s posted here, and in case you’ve missed it, we’ve mentioned:

You can reach out to help@postman.com if you’re currently blocked because of that limit (see here)
You can book some time to chat with one of our Product Manager if you’ve got more feedback to share about your specific use-case (see here)

@docking-module-part1 Here’s an excerpt from the blog announcing this change, is that the type of comment you were looking for?

To better align with the value Postman’s test automation offers for professional testers and quality engineers, we will make the following changes effective February 15, 2023, for free customers, and effective March 15, 2023, for new paid plan customers. Existing paid plan customers will see these limits effective upon plan renewal after March 15, 2023 (except Enterprise users, who have unlimited runs per month):

Local Collection Runner: Runs will be available with Free and Basic plans at 25 runs per month, Professional plan at 250 runs per month, and Enterprise plan at unlimited runs per month

Cloud-based Collection Runner: Runs will be billed on a consumption basis as part of the monitoring calls entitlement

Limits apply only to local runs via the Collection Runner, not via Newman or Postman CLI.

@alekdavis I think keeping these different public threads open, and the 1.5k+ feature requests from the community we’ve implemented are a better testimony of our open source philosophy. That said if you’d like to discuss the open philosophy feel free to open another thread as this is a bit off-topic here and I’d rather we keep the discussion focused on this specific change.

michaelderekjones · March 1, 2023, 9:49am

One of the key features which used to be free is now only really usable on the Enterprise plan.

So what you are saying is the value the test automation features offers is worth an extra 1200$ dollars per user, per year. That is the justification. You know that is more expensive that SoapUI?

250 runs on Professional considering its shared across the team will be used in no time at all for even smallish teams, so even Professional is of little use.

With SoapUI, there was always limits on some of the features. You need a paid for license to run the software in the CI, and for the reporting features, but it was always like that, so you knew where you stood.

The collection is akin to Test Suite, so you can’t even run a test suite in a tool designed for development and testing without having an Enterprise license.

I personally think you have shot yourself in the foot here. You’ve crippled the core product in what appears to be an attempt to push more Enterprise licenses.

I think you might find that you will lose out over time on all of the other tiers.

Users will try the free version first, hit the limit, and then go look at something else.

I would imagine that most of your Enterprise users try the free version first, and then for governance, require a fixed IP and the SSO consider Enterprise. I doubt anyone just jumps into an Enterprise license.

heamshrij · March 1, 2023, 3:36pm

I would not think that limiting the collection runs is a feature implementation request coming from community users who are using postman free licenses or otherwise. I also would think that notifying the user community about this limitation well ahead of time would have been decent way to implement this rather than surprising them , so the vast number of tests that are implemented under collections are not runnable in a user-friendly manner (I know there is the newman way to run). jsut my 2 cents

navigation-saganist2 · March 1, 2023, 4:02pm

Imposing a run limit on local collection runs in a workspace or sandbox may very well be the nail in the coffin for postman as a tool for my group. Anyone using postman’s collection runner to develop test flows for API testing will need to run collections over and over. It looks like postman is playing big brother here and setting limits on how many times I can “use postman as a tool” to help me get my job done. I strongly believe this is the wrong path for postman. Imagine how many developers would continue using vscode if there were limits on how many times developers could run their programs! Postman’s collection runner is used in the very same way . I’m extremely disappointed to see this change of approach.