Solr cluster

A step by step guide for a high availability Solr environment

# --> Got to /opt and download Solr cd /opt wget # --> Extract just the installation script from thee archive tar xzf solr-8.5.0.tgz solr-8.5.0/bin/ --strip-components=2 # --> Execute the script against the archive with some parameters # --> i -> Directory to extract the Solr installation archive; defaults to /opt/". The specified path must exist prior to using this script. # --> d -> Directory for live / writable Solr files, such as logs, pid files, and index data; defaults to /var/solr # --> u -> User to own the Solr files and run the Solr process as; defaults to solr. This script will create the specified user account if it does not exist. # --> s -> Service name; defaults to solr # --> p -> Port Solr should bind to; default is 8983 ./ solr-8.5.0.tgz -i /opt -d /var/solr -u solr -s solr -p 8983 Extracting solr-8.5.0.tgz to /opt Installing symlink /opt/solr -> /opt/solr-8.5.0 ... Installing /etc/init.d/solr script ... Installing /etc/default/ ... Service solr installed. Customize Solr startup configuration in /etc/default/ *** [WARN] *** Your open file limit is currently 1024.  It should be set to 65000 to avoid operational disruption.  If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or *** [WARN] ***  Your Max Processes Limit is currently 3894.  It should be set to 65000 to avoid operational disruption.  If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or Warning: Available entropy is low. As a result, use of the UUIDField, SSL, or any other features that require RNG might not work properly. To check for the amount of available entropy, use 'cat /proc/sys/kernel/random/entropy_avail'. Waiting up to 180 seconds to see Solr running on port 8983 [-] Started Solr server on port 8983 (pid=5500). Happy searching! Found 1 Solr nodes: Solr process 5500 running on port 8983 {   "solr_home":"/var/solr/data",   "version":"8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 - romseygeek - 2020-03-13 09:38:24",   "startTime":"2020-03-30T15:48:15.498Z",   "uptime":"0 days, 0 hours, 0 minutes, 13 seconds",   "memory":"140.5 MB (%27.4) of 512 MB"} [[email protected] opt]# service solr status Found 1 Solr nodes: Solr process 5500 running on port 8983 {   "solr_home":"/var/solr/data",   "version":"8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 - romseygeek - 2020-03-13 09:38:24",   "startTime":"2020-03-30T15:48:15.498Z",   "uptime":"0 days, 0 hours, 0 minutes, 59 seconds",   "memory":"145.5 MB (%28.4) of 512 MB"}
  • A Cluster is made up of one or more Solr Nodes, which are running instances of the Solr server process.
  • Each Node can host multiple Cores.
  • Each Core in a Cluster is a physical Replica for a logical Shard.
  • Every Replica uses the same configuration specified for the Collection that it is a part of.
  • The number of Replicas that each Shard has determines:
    • The level of redundancy built into the Collection and how fault tolerant the Cluster can be in the event that some Nodes become unavailable.
    • The theoretical limit in the number concurrent search requests that can be processed under heavy load.


When the collection is too large for one node, you can break it up and store it in sections by creating multiple shards. A Shard is a logical partition of the collection, containing a subset of documents from the collection, such that every document in a collection is contained in exactly one Shard. Which shard contains each document in a collection depends on the overall “Sharding” strategy for that collection. For example, you might have a collection where the “country” field of each document determines which shard it is part of, so documents from the same country are co-located. A different collection might simply use a “hash” on the uniqueKey of each document to determine its Shard.

Before SolrCloud, Solr supported Distributed Search, which allowed one query to be executed across multiple shards, so the query was executed against the entire Solr index and no documents would be missed from the search results. So splitting an index across shards is not exclusively a SolrCloud concept. There were, however, several problems with the distributed approach that necessitated improvement with SolrCloud:

  1. Splitting an index into shards was somewhat manual.
  2. There was no support for distributed indexing, which meant that you needed to explicitly send documents to a specific shard; Solr couldn’t figure out on its own what shards to send documents to.
  3. There was no load balancing or failover, so if you got a high number of queries, you needed to figure out where to send them and if one shard died it was just gone.

SolrCloud fixes all those problems. There is support for distributing both the index process and the queries automatically, and ZooKeeper provides failover and load balancing. Additionally, every shard can also have multiple replicas for additional robustness.

In SolrCloud there are no masters or slaves. Instead, every shard consists of at least one physical replica, exactly one of which is a leader. Leaders are automatically elected, initially on a first-come-first-served basis, and then based on the ZooKeeper process described at

If a leader goes down, one of the other replicas is automatically elected as the new leader.

When a document is sent to a Solr node for indexing, the system first determines which Shard that document belongs to, and then which node is currently hosting the leader for that shard. The document is then forwarded to the current leader for indexing, and the leader forwards the update to all of the other replicas.

Document Routing: Solr offers the ability to specify the router implementation used by a collection by specifying the  parameter when creating your collection.

If you use the (default) “” router, you can send documents with a prefix in the document ID which will be used to calculate the hash Solr uses to determine the shard a document is sent to for indexing. The prefix can be anything you’d like it to be (it doesn’t have to be the shard name, for example), but it must be consistent so Solr behaves consistently. For example, if you wanted to co-locate documents for a customer, you could use the customer name or ID as the prefix. If your customer is “IBM”, for example, with a document with the ID “12345”, you would insert the prefix into the document id field: “IBM!12345”. The exclamation mark (‘!’) is critical here, as it distinguishes the prefix used to determine which shard to direct the document to. If you do not want to influence how documents are stored, you don’t need to specify a prefix in your document ID.

Shard Splitting: When you create a collection in SolrCloud, you decide on the initial number shards to be used. But it can be difficult to know in advance the number of shards that you need, particularly when organizational requirements can change at a moment’s notice, and the cost of finding out later that you chose wrong can be high, involving creating new cores and re-indexing all of your data. The ability to split shards is in the Collections API. It currently allows splitting a shard into two pieces. The existing shard is left as-is, so the split action effectively makes two copies of the data as new shards. You can delete the old shard at a later time when you’re ready.

In most cases, when running in SolrCloud mode, indexing client applications should not send explicit commit requests. Rather, you should configure auto commits with  and auto soft-commits to make recent updates visible in search requests. This ensures that auto commits occur on a regular schedule in the cluster.

Ignoring commits from Client Applications: To enforce a policy where client applications should not send explicit commits, you should update all client applications that index data into SolrCloud. However, that is not always feasible, so Solr provides the , which allows you to ignore explicit commits and/or optimize requests from client applications without having refactored your client application code. To activate this request processor you’ll need to add the following to your :

 You can also configure it to just ignore optimize and let commits pass through by doing:

  1. Budget rental car pleasanton ca
  2. Wow roms
  3. Carbon health oakland, ca
  4. store


Note: We have a glossary covering collections, cores, shards, and replicas.

As you might imagine, every project is different. The size of the deployment (number of nodes, RAM and disk space) depends on the number of documents, the size of the stored fields, the number of collections, indexing frequency, query loading, and other factors that vary from one project to the next. The SearchStax Solutions team helps premium clients estimate these needs during the on-boarding process.

Deployment size is not cast in stone. Once the team has acquired some experience indexing and searching real data, it is easy to upgrade a deployment for more memory/disk or for additional Solr nodes (servers).

Specific best practices for nodes, shards, and replicas are presented below.

Best Practice: Use at least two nodes!

A single-node system cannot provide high-availability/fault-tolerant behavior. Production systems should have at least two nodes.

Best Practice: Use one shard!

Sharding splits up a huge index across multiple servers. It introduces a great deal of complexity to the system. Sharding multiplies the number of servers required to achieve high-availability/fault-tolerant behavior. Shards disable Managed Solr’s backup features. (Custom backups can be arranged for premium customers.)

If your index can fit comfortably on one server, then use one shard. This is Solr’s default behavior.

Best Practice: One replica per node!

To achieve high-availability/fault-tolerant behavior, every node of the cluster must have a replica of every collection. If some nodes are missing some replicas, there will be difficulties with backups and with Pulse monitoring of collections. A problem with a single node may take a collection out of service.

When you create the collection, set replicationFactor equal to the number of nodes in the cluster. Solr will automatically distribute the replicas to all nodes.

Java and the Lucene search library [6] form the basis for the search engine framework Apache Solr [1]. In the previous three articles, we set up Apache Solr on the soon-to-be-released Debian GNU/Linux 11 “Bullseye,” which initiated a single data core, uploaded example data, and demonstrated how to query output data in different ways and post-process it [2,3]. In part 3 [4], you have learned how to connect the relational database management system PostgreSQL [5] to Apache Solr and initiated a search in it.

The more documents you have to manage, the longer the answer time on a single-core setup. A multi-core Solr cluster helps to substantially reduce this answer time and increase the effectiveness of the setup. This article demonstrates how to do that and which traps to avoid.

Why and when taking clustering into account

To begin with, you need to understand what the term clustering stands for, why it is helpful to think about it, and especially when, how, and for who. There is no super-effective, all-inclusive recipe but several general criteria for the cluster setup that balance the load and help you keep your search engine’s answer time within a specific time range. This helps to run the search engine cluster reliably.

Generally speaking, the term clustering refers to a grouping of components that are similar to each other. Regarding Apache Solr, this means that you break down a large number of documents into smaller subsets based on the criteria you choose. You assign each subset to a single Apache Solr instance.

Instead of keeping all the documents in a single database, you store them in different topic-related databases or based on the letter range — for example, based on the first letter of the author’s last name. The first one goes from A to L and the second one from M to Z. To find information about books from Ernest Hemmingway, you have to look for them in the first database as the letter H is located alphabetically between A and L.

This setup already reduces your search area by 50% and, based on the assumption of an equally distributed number of book entries, reduces the search time likewise. In Apache Solr, this concept is called shard or slice, which describes a logical section of a single collection.

Someone who has only 500 documents can still easily handle the search based on a single core. In contrast, someone who has to manage a library of 100,000 documents needs a way to keep the response time within a certain level — if it takes too long, the provided service will not be used, and instead, the user will complain that searching takes way too long.

Also, the idealization is that two cores immediately reduce the search time by 50% and three cores by 66%, which is not true. The improvement is non-linear and about 1.5 (two cores) to 1.2 (three to four cores in a cluster). This non-linear improvement is known as Amdahl’s Law [7]. The additional time comes from the overhead needed to run the single cores, coordinate the search processes, and manage its results. In general, there is a remarkable improvement, but non-linear and only up to a certain point. In certain circumstances, even five or more parallel cores already form the boundary and have the same response time as four cores but require remarkably more resources than hardware, energy, and bandwidth.

Clustering in Apache Solr in more detail

So far, our Solr-based search engine consists of only a single node or core. The next level is to run more than one node or core in parallel to process more than one search request at a time.

A Solr cluster is a set of single Solr nodes. Also, a cluster itself can contain many document collections. The architectural principle behind Solr is non-master-slave. As a result, every Solr node is a master of its own.

The first step towards fault tolerance and higher availability is running a single Solr instance as separate processes. For the coordination between the different operations, Apache Zookeeper [8] comes into play. ZooKeeper describes itself as “a centralized service for maintaining configuration information, naming, providing distributed synchronization and providing group services.”

To go even more significantly, Apache Solr includes the ability to set up an entire cluster of various Solr servers called SolrCloud [9]. Using SolrCloud, you can profit from distributed indexing and search capabilities designed to handle an even more significant number of indexed documents.

Run Apache Solr with more than a single core as a collection

As already described in part 1 of this article series [2], Apache Solr runs under the user solr. The project directory under /opt/solr-8.7.0 (adjust the version number according to the Apache Solr version you use) and the variable data directory under /var/solr must belong to the solr user. If not done yet, you can achieve this as the root user with the help of these two commands:

# chmod -R solr:solr /var/solr

# chmod -R solr:solr /opt/solr-8.7.0

The next step is starting Apache Solr in cloud mode. As user solr, run the script in the following way:

$ bin/solr -e cloud

With this command, you start an interactive session to set up an entire SolrCloud cluster with embedded ZooKeeper. First, specify how many nodes the Solr cluster should consist of. The range is between 1 and 4, and the default value is 2:

Welcome to the SolrCloud example!

This interactive session will help you launch a SolrCloud cluster on your local workstation.
To begin, how many Solr nodes would you like to run in your local cluster? (specify 1-4 nodes)[2]

Next, the script bin/solr prompts you for the port to bind each of the Solr nodes to. For the 1st node, it suggests port #8983, and for the 2nd node the port #7574 as follows:

Please enter the port for node1 [8983]

Please enter the port for node2 [7574]

You can choose any available port here. Please make sure beforehand that other network services are not yet using the specified ports. However, at least for the example used here, it is recommended to keep the default values. After answering the question, the script bin/solr starts the individual nodes one by one. Internally, it executes the following commands:

$ bin/solr start -cloud-s example/cloud/node1/solr -p8983

$ bin/solr start -cloud-s example/cloud/node2/solr -p7574

The figure below demonstrates this step for the first node. The output of the second node is likewise.

Simultaneously, the first node will also start an embedded ZooKeeper server. This server is bound to port #9983. The example call above the Solr home for the first node is the directory example/cloud/node1/solr as indicated by the -s option. The figure below shows the corresponding status messages.

Having started the two nodes in the cluster, the script will ask you for some more information — the name of the collection to create. The default value is getting started that we substitute by cars from part 2 of this article series [3] here:

Please provide a name for your new collection: [gettingstarted] cars

This entry is similar to the following script call that allows you to create the document collection cars individually:

$ bin/solr create_collection -c cars

Finally, the script prompts you for the number of shards and the number of replicas per shard. For this case, we stick to the default values of 2 shards and 2 replicas per shard. This allows you to understand how a collection is distributed across multiple nodes in a SolrCloud cluster, and SolrCloud handles the replication feature.

Now their Solr Cluster is up and running and ready to go. There are several changes in the Solr Administration panel, like additional menu entries for cloud and collections. The three figures below show the information that is available about the previously created cloud. The first image displays the node state and its current usage.

The second image displays the organization of the cloud as a directed graph. Each active node is green with its name, IP address, and port number as previously defined. You find this information under the menu entry Cloud and in the submenu Graph.

The third image displays information about the collection of cars as well as its shards and replicas. To see the details for the collection, click on the menu entry “cars” that is located right of the main menu and below the button “Add Collection.” The corresponding shard information becomes visible if you click on the bold text labeled “Shard: shard1” and “Shard2”.

Apache Solr also provides information on the command line. For this purpose, it offers the subcommand healthcheck. As additional parameters, enter -c followed by the name of the collection. In our case, the command is as follows to run the check on the cars collection:

$ bin/solr healthcheck -c cars

The information is returned as a JSON file and shown below.

As explained in the Solr manual, the healthcheck command collects basic information about each replica in a collection. This covers the number of Documents, its current status like active or down, and the address — where the replica is located in the SolrCloud. Finally, you can now add Documents to SolrCloud. The call below adds the XML files to the cluster that are stored in the directory datasets/cars:

$ bin/post -c cars datasets/cars/*.xml

The uploaded data is distributed to the different cores and ready to be queried from there. See the previous articles on how to do that.


Apache Solr is designed to handle a large number of data sets. To minimize the answer time, run Solr as a cluster, as explained before. It needs a few steps, but we think it is worth having happier users of your document storage.

About the authors

Jacqui Kabeta is an environmentalist, avid researcher, trainer, and mentor. In several African countries, she has worked in the IT industry and NGO environments.

Frank Hofmann is an IT developer, trainer, and author and prefers to work from Berlin, Geneva, and Cape Town. Co-author of the Debian Package Management Book available from

Thank you

The authors would like to thank Saif du Plessis for his help while preparing the article.

Links and References

  • [1] Apache Solr,
  • [2] Frank Hofmann and Jacqui Kabeta: Introduction to Apache Solr. Part 1,
  • [3] Frank Hofmann and Jacqui Kabeta: Introduction to Apache Solr. Part 2: Querying Solr. Part 2,
  • [4] Frank Hofmann and Jacqui Kabeta: Introduction to Apache Solr. Part 3: Connecting PostgreSQL and Apache Solr,
  • [5] PostgreSQL,
  • [6] Lucene,
  • [7] Amdahl’s Law, Wikipedia,
  • [8] Zookeeper,
  • [9] SolrCloud,

Cluster solr

Only when he wants to. And the idea of how this can be organized quickly comes to his mind, tired of a Friday working day. He sits in his office in a black cloth chair.

However, it still didn't stop As it was decided, the next day, in the late afternoon, Anna and Marina went out for a walk. The weather was wonderful, really summer and both dressed appropriately lightly, sexy and beautiful. Anna is in a short black dress, with wide straps forming a low neckline crossing in the bodice.

And Marina in white tight-fitting trousers and a blue sleeveless button-down blouse with a small collar and an open back.

You will also like:

Gentle lips cover the head, the tongue gets down to business. Slowly going down the trunk, she sinks my pride further and further. Already my throat is gently massaging me.

1384 1385 1386 1387 1388