What is the recommended topology for
We strongly recommend running an
nsqd alongside any service(s) that produce
nsqd is a relatively lightweight process with a bounded memory footprint,
which makes it well suited to “playing nice with others”.
This pattern aids in structuring message flow as a consumption problem rather than a production one.
Another benefit is that it essentially forms an independent, sharded, silo of data for that topic on a given host.
NOTE: this isn’t an absolute requirement though, it’s just simpler (see question below).
nsqlookupd be used by producers to find where to publish to?
NSQ promotes a consumer-side discovery model that alleviates the upfront configuration burden of having to tell consumers where to find the topic(s) they need.
However, it does not provide any means to solve the problem of where a service should publish to. This is a chicken and egg problem, the topic does not exist prior to the first publish.
nsqd (see question above), you sidestep this problem entirely
(your service simply publishes to the local
nsqd) and allow NSQ’s runtime
discovery system to work naturally.
I just want to use
nsqd as a work queue on a single node, is that
a suitable use case?
nsqd can run standalone just fine.
nsqlookupd is beneficial in larger distributed environments.
nsqlookupd should I run?
Typically only a few depending on your cluster size, number of
nodes and consumers, and your desired fault tolerance.
3 or 5 works really well for deployments involving up to several hundred hosts and thousands of consumers.
nsqlookupd nodes do not require coordination to answer queries. The
metadata in the cluster is eventually consistent.
Do I need a client library to publish messages?
NO! Just use the HTTP endpoints for publishing (
simple, it’s easy, and it’s ubiquitous in almost any programming environment.
In fact, the overwhelming majority of NSQ deployments use HTTP to publish.
Why force a client to handle responses to the TCP protocol’s
We believe NSQ’s default mode of operation should prioritize safety and we wanted the protocol to be simple and consistent.
When can a
nsqdis in the middle of cleanly exiting.
(1) and (2) should be considered programming errors. (3) and (4) are rare and (5) is a natural part of any TCP based protocol.
How can I mitigate scenario (3) above?
Deleting topics is a relatively infrequent operation. If you need to delete a topic, orchestrate the timing such that publishes eliciting topic creations will never be performed until a sufficient amount of time has elapsed since deletion.
How do you recommend naming topics and channels?
A topic name should describe the data in the stream.
A channel name should describe the work performed by its consumers.
For example, good topic names are
page_views and good channel names are
Are there any limitations to the number of topics and channels a single
nsqd can support?
There are no built-in limits imposed. It is only limited by the memory
and CPU of the host
nsqd is running on (per-client CPU usage was greatly
reduced in issue #236).
How are new topics announced to the cluster?
SUB to a topic will create the topic on an
metadata will then propagate to the configured
nsqlookupd. Other readers
will discover this topic by periodically querying the
Can NSQ do RPC?
Yes, it’s possible, but NSQ was not designed with this use case in mind.
We intend to publish some docs on how this could be structured but in the meantime reach out if you’re interested.
Why are you forcing me to use Tornado?
pynsq was originally intended as a consumer-focussed library and consuming
the NSQ protocol is far simpler to implement in Python with an asynchronous
framework (notably due to NSQ’s push oriented protocol).
Tornado’s API is simple and performs reasonably well.
Is the Tornado IOLoop required to publish?
nsqd exposes HTTP endpoints (
/mpub) for dead-simple
programming language agnostic publishing.
If you’re worried about the overhead of HTTP, don’t be. Additionally,
/mpub reduces the overhead of HTTP by publishing in bulk (atomically!).
When would I want to use
When high performance and low overhead is a priority.
Writer uses the TCP protcol’s
MPUB commands which have less
overhead compared to their HTTP counterparts.
What if I just want to “fire and forget” (I can tolerate message loss!)?
Writer and don’t specify a callback to the publish method.
NOTE: this only has the effect of simpler client side code, behind the scenes
pynsq still has to handle the response from
nsqd (i.e. there isn’t
a performance benefit of doing this).
Special thanks to Dustin Oprea (@DustinOprea) for kickstarting this FAQ.