GetTwitter

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

GetTwitter

DenisP
I'm trying to use get GetTwitter processor.

I've created all what is needed on the twitter side (consumer keys, tokens,
access token,...)
These information run correctly when I use them from other technologies Than
wifi (i.e. I can retrieve information from twitter)

But when I run my controller twitter's response to the GetTwitter controller
is systematically "HTTP_ERROR: HTTP/1.1 420 Enhance Your Calm. Will attempt
to reconnect"

Reading the twitter's app developper documentation, it is said that such an
error is due to "Returned when an application is being rate limited ."

 the API is not "rate limited" from other technologies, and is "rate
limited" from this processor...
Can't understand....



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: GetTwitter

Pierre Villard
Hi Denis,

Twitter API has limitations regarding how many calls you can perform over
time.
By default the processor will be executed as often as possible by NiFi
(thousands per second).
I'd suggest to change the "run schedule" parameter in the scheduling
configuration of the processor. Something like once every 60 seconds seems
reasonable.

Pierre

2017-08-31 14:57 GMT+02:00 DenisP <[hidden email]>:

> I'm trying to use get GetTwitter processor.
>
> I've created all what is needed on the twitter side (consumer keys, tokens,
> access token,...)
> These information run correctly when I use them from other technologies
> Than
> wifi (i.e. I can retrieve information from twitter)
>
> But when I run my controller twitter's response to the GetTwitter
> controller
> is systematically "HTTP_ERROR: HTTP/1.1 420 Enhance Your Calm. Will attempt
> to reconnect"
>
> Reading the twitter's app developper documentation, it is said that such an
> error is due to "Returned when an application is being rate limited ."
>
>  the API is not "rate limited" from other technologies, and is "rate
> limited" from this processor...
> Can't understand....
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: GetTwitter

Mark Payne
In reply to this post by DenisP
Denis,

There are two common problems that will cause this error. Firstly, if you use the same
consumer key, token, etc. with two different applications - or with 2 different GetTwitter
processors then you'll run into this issue. So if you have multiple GetTwitter processors,
for some reason, then they will each need their own credentials.

The second issue that I've seen cause this is if you are running a NiFi cluster. In this case,
you will want to ensure that GetTwitter is only running on the Primary Node (this is configured
in the Scheduling tab of the Processor Configuration). Once this is done, you may still get the
warning when the Processor is started due to a problem with how we handle processor initialization
(there is a JIRA to address this) but the warning should only occur once and it should then
continue working.

Do either of the above issues appear to be applicable to your use case?

Thanks
-Mark


> On Aug 31, 2017, at 8:57 AM, DenisP <[hidden email]> wrote:
>
> I'm trying to use get GetTwitter processor.
>
> I've created all what is needed on the twitter side (consumer keys, tokens,
> access token,...)
> These information run correctly when I use them from other technologies Than
> wifi (i.e. I can retrieve information from twitter)
>
> But when I run my controller twitter's response to the GetTwitter controller
> is systematically "HTTP_ERROR: HTTP/1.1 420 Enhance Your Calm. Will attempt
> to reconnect"
>
> Reading the twitter's app developper documentation, it is said that such an
> error is due to "Returned when an application is being rate limited ."
>
> the API is not "rate limited" from other technologies, and is "rate
> limited" from this processor...
> Can't understand....
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Reply | Threaded
Open this post in threaded view
|

Re: GetTwitter

DenisP
In reply to this post by Pierre Villard
I changed the  "run schedule to 60 sec" - Same problem...



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: GetTwitter

Aldrin Piri
Once you are rate limited for the provided API keys, you also need to
disable the processor for a period of time (not sure of the exact duration,
but typically 10+ minutes) or generate a new set of credentials. Otherwise,
the clock never resets for the rate limiting.

On Thu, Aug 31, 2017 at 9:42 AM, DenisP <[hidden email]> wrote:

> I changed the  "run schedule to 60 sec" - Same problem...
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: GetTwitter

DenisP
In reply to this post by Mark Payne
I have tested these consumers keys,... with spark-scale in order to check my
information are correct (and they are).

I only have One GetTwitter processor, and it is not possible to make It run:

2017-08-31 15:44:34,715 INFO [hosebird-client-io-thread-0]
com.twitter.hbc.httpclient.ClientBase
GetTwitter[id=015e104c-e4d7-103c-3f6e-151b88969592] *Establishing a
connection*
2017-08-31 15:44:36,239 WARN [hosebird-client-io-thread-0]
com.twitter.hbc.httpclient.ClientBase
GetTwitter[id=015e104c-e4d7-103c-3f6e-151b88969592] *Error connecting w/
status code - 420, reason - Enhance Your Calm*
2017-08-31 15:44:36,239 INFO [hosebird-client-io-thread-0]
com.twitter.hbc.httpclient.ClientBase
GetTwitter[id=015e104c-e4d7-103c-3f6e-151b88969592] *exit event - Retries
exhausted*

My NIFI is standalone, no cluster...




--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/