Posting tweets with Twitter API 1.1 returns error for punctuation characters

[floripare]

Hello community!

I’m not a developer, but I’ve managed to create a Postman monitor that publishes every few hours a tweet using the Twitter API v1.1. It’s working, and it’s running, yay! A pre-request script fetches the content from a JSON file on my website that is updated hourly and has only a key/value pair:

{"tweet_text":"Hello, world!"}

The value is then stored in a variable that is passed on to Twitter API as being the content for the “status” parameter.

The problem is that it only works when the content does not contain any of the following characters:

: / ? ! ( ) + #

Attempts to post a tweet containing any of the above characters return an error code 32, “Could not authenticate you” from the Twitter API. (So the very “Hello world!” example given above would not work, given the exclamation point ending the sentence. And of course being unable to use the proper punctuation, let alone the # symbols, makes for poor tweets. (For the purpose of testing, I set the string on the pre-request script itself, instead of fetching from my site):

Any ideas on what may be causing my requests to result in errors?

Thanks
Edit: I’ve already tried

• escaping using \ followed by 0x3A and :, \u003A.
• using the escape() object
• sending the parameters as part of the body (x-www-form-urlencoded option)
• EncodeURI and EncodeURIComponent
• backslash before each punctuation mark

[bpricilla]

Hello @floripare

Can you try using the encodeURI() function.

const URIEncodedString = encodeURIComponent(tweet_text);
console.log(URIEncodedString);

Which version of postman are you using? Seems like there’s a open issue regarding the encoding.

Even this is not working, then you can try encoding the entire URL and send?

[floripare]

Hi @bpricilla, thanks for replying!

I tried your suggestion, but Twitter API returned the same error 32.

As a matter of fact, I now recall having already tried both EncodeURI and EncodeURIComponent. (Sorry, I forgot to mention, will edit my initial post.)

When there’s one of the punctuation marks, it doesn’t work, and when there isn’t, the string is encoded twice, as I have the setting Encode URL automatically ON.

Now notice how the encoded string does not encode the ' and ! signs (However, even when I removed them from the string, it still returns the 32 error)

Original: This is a test: Let's see how it goes! Will it go through? +plus #hashtagsarefun

Encoded: This%20is%20a%20test%3A%20Let's%20see%20how%20it%20goes!%20Will%20it%20go%20through%3F%20%2Bplus%20%23hashtagsarefun

I’ve tried escaping each of these characters with a backslash, but it didn’t work either.

[bpricilla]

Oh I get that Just to be sure the authentication is working,

so the tweets without special characters is working right? Once we confirm that, we can say then the issue with the malformation of the request.

[floripare]

Yes, the authentication is working normally. Any tweets without those characters are published and return a 200.

All I could find on the Twitter API documentation is that they require that URLs be percent encoded and comply with RFC 3986.

[floripare]

Hi @bpricilla, good morning!

To further test how strings are being encoded and/or escaped, I run a POST request with the Postman API. The result clearly shows that the + sign before the word “plus” vanishes, and so do the # sign and the following hashtag.

Should I open a ticket or do you think this is the same issue being investigated on that GitHub issue you linked to?

[bpricilla]

Hi @floripare

Yes you are right, anything after “#” disappears. Since the parent issue is related to URL encoding, shall we try adding your use case also to the same ticket as new comment?

[floripare]

@bpricilla

shall we try adding your use case also to the same ticket as new comment?

Please feel free to do so. Since I’m not an IT person, I don’t feel like doing it myself because every time I tried raising GH issues in the past I end up receiving replies that I have a hard time understanding

[floripare]

I’ve opened a ticket on this issue last night (#68068). What’s happening is that Postman is eliminating several punctuation characters, and not encoding the # sign. So unless Postman requires that these characters are escaped, there must be some thing at play before the request is sent to Twitter API.

For this text:

"This is a test: Let's see how it goes! Will it go through? +plus = #hashtagsarefun"

Postman is generating this URL:

https://api.twitter.com/1.1/statuses/update.json?status=This%20is%20a%20test%20Lets%20see%20how%20it%20goes%20Will%20it%20go%20through%20plus%20#hashtagsarefun

As you can see, no only the punctuation is not being %-encoded, they are simply vanishing. And the # is not being encoded.

[bpricilla]

Okay I get that I hope they respond back asap to you

[udit.vasu]

@floripare Postman v8 follows the WHATWG (URL) standard instead of RFC-3986 for encoding the URL components.

Refer this comment for more details. You can achieve RFC-3986 behavior by adding the following pre-request script at collection or request level:

pm.request.url.query.each((q) => {
     q.update(encodeURIComponent(q.toString()));
});

[floripare]

Thanks for chipping in, @udit.vasu.

I was about to update this thread to say I couldn’t find a definite solution, but managed this Sunday to find a workaround that allowed me to post several special characters. With it, I still can’t post any tweets with ! ( ) ' * and perhaps other characters, but was able to include# @ ? , by:

• removing the status parameter,
• setting the body to x-www-form-urlencoded type,
• adding the status parameter to the body

BTW, the value of the status parameter comes from a variable. The text is fetched from an external source, then set in a environment variable as {{tweet_text}}, in a initial pre-request script:

pm.sendRequest("https://example.com/tweet_text.json", function (err, response) {
console.log(response.json());

const tweet_text = Object.values(response.json());
pm.environment.set("tweet_text", tweet_text) ;

});

Now, I’ve just tried your proposed solution, by reverting my request to include the status as a parameter.

However, what happens is Postman will then send ?status%3D%7B%7Btweet_test%7D%7D.

Instead of resolving the variable {{tweet_text}}, it encodes the brackets and passes them on.

Any idea of what I could try?

[udit.vasu]

We have to resolve the variables in the URL before encoding the query params.

I hope the following pre-request script will work for you:

let url = pm.request.url;

// resolve variables in the URL
url.update(pm.variables.replaceIn(url.toString()));

// encode query parameters
url.query.each((q) => {
     q.update(encodeURIComponent(q.toString()));
});

[floripare]

Thank you again @udit.vasu

This somehow encodes the = (not sure if this matters), but does not encode ' :

I believe the problem is that Twitter API needs that all characters that are

• neither alphanumeric
• nor - _ . ~

to be encoded, while encodeURIComponent() does not seem to encode

• ! ' * ( )

and other punctuation/special characters.