What are the core architectural differences between these technologies?  

Also, what use cases are generally more appropriate for each?



Solr vs. ElasticSearch

I&#39;m currently looking at other search methods rather than having a huge SQL query.
I saw [elasticsearch][1] recently and played with [whoosh][2] (a Python implementation of a search engine).

Can you give reasons for your choice(s)?


  [1]: http://elasticsearch.org/
  [2]: http://whoosh.ca/ &quot;Whoosh&quot;

ElasticSearch, Sphinx, Lucene, Solr, Xapian. Which fits for which usage?

I&#39;m building a Django site and I am looking for a search engine.

A few candidates:

- Lucene/Lucene with Compass/Solr

- Sphinx

- Postgresql built-in full text search

- MySQl built-in full text search

Selection criteria:

- result relevance and ranking
- searching and indexing speed
- ease of use and ease of integration with Django
- resource requirements - site will be hosted on a [VPS][1], so ideally the search engine wouldn&#39;t require a lot of RAM and CPU
- scalability
- extra features such as &quot;did you mean?&quot;, related searches, etc

Anyone who has had experience with the search engines above, or other engines not in the list -- I would love to hear your opinions.

EDIT: As for indexing needs, as users keep entering data into the site, those data would need to be indexed continuously. It doesn&#39;t have to be real time, but ideally new data would show up in index with no more than 15 - 30 minutes delay

  [1]: http://en.wikipedia.org/wiki/Virtual_private_server


Comparison of full text search engine - Lucene, Sphinx, Postgresql, MySQL?

With the NoSQL movement growing based on document-based databases, I&#39;ve looked at MongoDB lately.  I have noticed a striking similarity with how to treat items as &quot;Documents&quot;, just like Lucene does (and users of Solr).

So, the question: **Why would you want to use NoSQL (MongoDB, Cassandra, CouchDB, etc) over Lucene (or Solr) as your &quot;database&quot;?**

What I am (and I am sure others are) looking for in an answer is some deep-dive comparisons of them.  Let&#39;s skip over relational database discussions all together, as they serve a different purpose. 

Lucene gives some serious advantages, such as powerful searching and weight systems.  Not to mention facets in Solr (which Solr is being integrated into Lucene soon, yay!).  You can use Lucene documents to store IDs, and access the documents as such just like MongoDB.  Mix it with Solr, and you now get a WebService-based, load balanced solution.

You can even throw in a comparison of out-of-proc cache providers such as Velocity or MemCached when talking about similar data storing and scalability of MongoDB.

The restrictions around MongoDB reminds me of using MemCached, but I can use Microsoft&#39;s Velocity and have more grouping and list collection power over MongoDB (I think).  Can&#39;t get any faster or scalable than caching data in memory.  Even Lucene has a memory provider.
 
MongoDB (and others) do have some advantages, such as the ease of use of their API.  New up a document, create an id, and store it.  Done.  Nice and easy.



NoSQL (MongoDB) vs Lucene (or Solr) as your database

I&#39;m looking for a stand-alone full-text search server with the following properties:

* Must operate as a stand-alone server that can serve search requests from multiple clients
* Must be able to do &quot;bulk indexing&quot; by indexing the result of an SQL query: say &quot;SELECT id, text_to_index FROM documents;&quot;
* Must be free software and must run on Linux with MySQL as the database
* Must be fast (rules out MySQL&#39;s internal full-text search)

The alternatives I&#39;ve found that have these properties are:

* Solr (based on Lucene)
* ElasticSearch (also based on Lucene)
* Sphinx

My questions:

* How do they compare? 
* Have I missed any alternatives?
* I know that each use case is different, but are there certain cases where I would definitely *not* want to use a certain package?


Choosing a stand-alone full-text search server: Sphinx or SOLR?

I know ElasticSearch is built upon Apache Lucene but I want to know the significant differences between the two.

What is the difference between Lucene and Elasticsearch

I know that Lucene and Solr are 2 differents [Apache projects][1] that are made to work together, but I don&#39;t understand what is the aim of each project.

For what I understood until now is that Lucene is used to create a search index and Solr use this index to perform searches.
Am I right or is this a totally different approach?


  [1]: http://lucene.apache.org/

Difference between solr and lucene

I am learning NoSQL and looking at different options for one of my client&#39;s requirements.  I have gone through various resources before putting up this question (a person with little knowledge in NoSQL)

 - I need to store data at faster rate and read data. 
 - Fully fail-safe and easily scalable.
 - Able to search through data for Analytics.

I ended up with a short list of: `Cassandra and Elasticsearch`

What I do understand is Cassandra is a perfect NoSQL storage solution for me, as I can write data and read data using indexes.  Where it fails or it could fail is on Analytics. In the future, if I want to get data from `from_date to to_date`, or more ways to get data for analytics, if I don&#39;t design the Data model properly or keeping long term sight, which might be quite hard in ever changing world.

While `Elastic Search` is best at indexing (backed by Lucene), and can search the data randomly by throwing some random text. But does it work the same for even if I want to retrieve data `from_date to to_date` (I expect it might be). But the real question is, is it a Search Engine, or perfect NoSQL data storage like Cassandra? If yes, why do we still need Cassandra?

If both of these are in different world, please explain that! How do we combine them to get a more effective solution?

Elasticsearch vs Cassandra vs Elasticsearch with Cassandra

I have a large solr index, and I have noticed some fields are not updated correctly (the index is dynamic).

This has resulted in some fields having an empty &quot;id&quot; field.

I have tried these queries, but they didn&#39;t work:

     id:&#39;&#39;
     id:NULL
     id:null
     id:&quot;&quot;
     id:
     id:[&#39;&#39; TO *]

Is there a way to query empty fields?

Thanks

How to query SOLR for empty fields?

I read some document about Lucene; also I read the document in this link
([http://lucene.sourceforge.net/talks/pisa][1]).

I don&#39;t really understand how Lucene indexes documents and don&#39;t understand which algorithms Lucene uses for indexing?

On the above link, it says Lucene uses this algorithm for indexing:

&gt;  - incremental algorithm:
&gt;   - maintain a stack of segment indices
&gt;   - create index for each incoming document
&gt;   - push new indexes onto the stack
&gt;   - let b=10 be the merge factor; M=8

---

    for (size = 1; size &lt; M; size *= b) {
    	if (there are b indexes with size docs on top of the stack) {
    		pop them off the stack;
    		merge them into a single index;
    		push the merged index onto the stack;
    	} else {
    		break;
    	}
    }

How does this algorithm provide optimized indexing?

Does Lucene use B-tree algorithm or any other algorithm like that for indexing 
- or does it have a particular algorithm?

  [1]: http://lucene.sourceforge.net/talks/pisa

How does lucene index documents?

Can someone please explain the difference between the different analyzers within Lucene?  I am getting a maxClauseCount exception and I understand that I can avoid this by using a KeywordAnalyzer but I don&#39;t want to change from the StandardAnalyzer without understanding the issues surrounding analyzers.  Thanks very much.

Comparison of Lucene Analyzers

I would like to find out how lucene search works so fast. I can&#39;t find any useful docs on the web. If you have anything (short of lucene source code) to read, let me know.

A text search query using mysql5 text search with index takes about 18 minutes in my case. A lucene search for the same query takes less than a second.

How does Lucene work

I&#39;m working on a solr query similar to the following:

    ((myField:superneat AND myOtherField:somethingElse) OR NOT myField:superneat)

When running this, no results are returned.  Using criteria on either side of the OR NOT returns results that I&#39;d expect - they are just not working well together.  In the case that **myField** matches *superneat*, I&#39;m intending to also ensure that **myOtherField** is set to *somethingElse*, but if **myField** is not *superneat*, include it in the results.

Can someone explain why solr is not returning results for this kind of query?  Should the query be restructured somehow - or is there a different way in which solr can be used to achieve the desired result?

using OR and NOT in solr query

Even though I read the documentation for Elasticsearch to understand what a percolator is. I still have difficulty understanding what it means and where it is used in simple terms. Can anyone provide me with more details?

What does percolator mean/do in elasticsearch?

I&#39;m trying to use Apache Lucene for tokenizing, and I am baffled at the process to obtain Tokens from a `TokenStream`.

The worst part is that I&#39;m looking at the comments in the JavaDocs that address my question.

http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/analysis/TokenStream.html#incrementToken%28%29

Somehow, an `AttributeSource` is supposed to be used, rather than `Token`s.  I&#39;m totally at a loss.  

Can anyone explain how to get token-like information from a TokenStream?

How to get a Token from a Lucene TokenStream?

I read https://stackoverflow.com/questions/468405/lucene-net-how-to-incorporate-multiple-fields-in-queryparser but i didn&#39;t get it.

At the moment i have a very strange construction like:

    parser = New QueryParser(&quot;bodytext&quot;, analyzer)
    parser2 = New QueryParser(&quot;title&quot;, analyzer)
    query = parser.Parse(strSuchbegriff)
    query2 = parser.Parse(strSuchbegriff)

What can i do for something like:

    parser = New QuerParser (&quot;bodytext&quot; , &quot;title&quot;,analyzer)
    query =parser.Parse(strSuchbegriff) 

so the Parser looks for the searching word in the field &quot;bodytext&quot; an in the field &quot;title&quot;.

How to specify two Fields in Lucene QueryParser?

I recently switched from Postgres to Solr and saw a ~50x speed up in our queries. The queries we run involve multiple ranges, and our data is vehicle listings. For example: &quot;Find all vehicles with mileage &lt; 50,000, $5,000 &lt; price &lt; $10,000, make=Mazda...&quot;

I created indices on all the relevant columns in Postgres, so it should be a pretty fair comparison. Looking at the query plan in Postgres though it was still just using a single index and then scanning (I assume because it couldn&#39;t make use of all the different indices).

As I understand it, Postgres and Solr use vaguely similar data structures (B-trees), and they both cache data in-memory. So I&#39;m wondering where such a large performance difference comes from.

What differences in architecture would explain this?

Why is Solr so much faster than Postgres?

All of us have come across the recent hype of no-SQL solutions lately. MongoDB, CouchDB, BigTable, Cassandra, and others have been listed as no-SQL options. Here&#39;s an example:

http://architects.dzone.com/articles/what-nosql-store-should-i-use

However, three years ago a co-worker and I were using Lucene.NET as what seem to fit the description of no-SQL. We did not use it just for user-inputted search queries; we used it to make a few reindexed RDBMS table data extremely performant. We implemented our own .NET sort-of-equivalent-to-Solr service to manage these indexes and make them callable. When I left the company, the team switched to Solr itself. (For those not in the know, Solr is a web service that wraps Lucene with REST-callable queries and index dumps.)

What I don&#39;t understand is, why is Solr not counted in the typical lists of no-SQL solution options? Am I missing something here? I assume that there are technical reasons why Solr is not comparable to the likes of CouchDB, etc., and in fact I understand that CouchDB uses Lucene as its data store (yes?), but what disqualifies Solr? 

I&#39;m not asking as some kind of Solr fanboy or anything, I just don&#39;t understand why Solr and the like don&#39;t fit the definition of no-SQL, and if Solr technically does fit the definition then what about it likely makes people pooh-pooh it? I&#39;m asking because I&#39;m having difficulty determining whether I should continue using Lucene-based solutions (like Solr) for solutions that I build or if I should really do more research with these other options.

Why are document stores like Lucene / Solr not included in NoSQL conversations?

I&#39;ve [asked a simlar question on Meta Stack Overflow][1], but that deals specifically with whether or not [Lucene.NET][2] is used on Stack&amp;nbsp;Overflow.

The purpose of the question here is more of a hypotetical, as to what approaches one would make if they were to use Lucene.NET as a basis for in-site search and other factors in a site *like* Stack&amp;nbsp;Overflow [SO]. 

As per the entry on the Stack&amp;nbsp;Overflow blog titled &quot;[SQL 2008 Full-Text Search Problems][3]&quot; there was a **strong** indication that Lucene.NET was being considered at some point, but it appears that is definitely not the case, as per the comment by [Geoff Dalgas][4] on February 19th 2010:

&gt; Lucene.NET is not being used for Stack
&gt; Overflow - we are using SQL Server
&gt; Full Text indexing. Search is an area
&gt; where we continue to make minor
&gt; tweaks.

So my question is, how would one utilize Lucene.NET into a site which has the same semantics of Stack&amp;nbsp;Overflow?

Here is some background and what I&#39;ve done/thought about so far (yes, I&#39;ve been implementing most of this and search is the last aspect I have to complete):

**Technologies:**

- [ASP.NET MVC][5]
- [SQL Server 2008][6]
- [.NET 3.5][7]
- [C# 3.0][8]

And of course, the star of the show, Lucene.NET.

The intention is also to move to .NET/C# 4.0 ASAP. While I don&#39;t think it&#39;s a game-changer, it should be noted.

Before getting into aspects of Lucene.NET, it&#39;s important to point out the SQL Server 2008 aspects of it, as well as the models involved.

**Models**

This system has more than one primary model type in comparison to Stack&amp;nbsp;Overflow. Some examples of these models are:

- Questions: These are questions that people can ask. People can reply to questions, just like on Stack&amp;nbsp;Overflow.
- Notes: These are one-way projections, so as opposed to a question, you are making a statement about content.  People can&#39;t post replies to this.
- Events: This is data about a real-time event.  It has location information, date/time information.

The important thing to note about these models:

- They all have a Name/Title (text) property and a Body (HTML) property (the formats are irrelevant, as the content will be parsed appropriately for analysis).
- Every instance of a model has a unique URL on the site

Then there are the things that Stack&amp;nbsp;Overflow provides which IMO, are decorators to the models. These decorators can have different cardinalities, either being one-to-one or one-to-many:

- Votes: Keyed on the user
- Replies: Optional, as an example, see the Notes case above
- Favorited: Is the model listed as a favorite of a user?
- Comments: (optional)
- Tag Associations: Tags are in a separate table, so as not to replicate the tag for each model.  There is a link between the model and the tag associations table, and then from the tag associations table to the tags table.

And there are supporting tallies which in themselves are one-to-one decorators to the models that are keyed to them in the same way (usually by a model id type and the model id):

- Vote tallies: Total postive, negative votes, [Wilson Score interval][9] (this is important, it&#39;s going to determine the confidence level based on votes for an entry, for the most part, assume the lower bound of the Wilson interval).

Replies (answers) are models that have most of the decorators that most models have, they just don&#39;t have a title or url, and whether or not a model has a reply is optional.  If replies are allowed, it is of course a one-to-many relationship.

**SQL Server 2008**

The tables pretty much follow the layout of the models above, with separate tables for the decorators, as well as some supporting tables and views, stored procedures, etc.

It should be noted that the decision to not use full-text search is based primarily on the fact that it doesn&#39;t normalize scores like Lucene.NET. I&#39;m open to suggestions on how to utilize text-based search, but I will have to perform searches across multiple model types, so keep in mind I&#39;m going to need to normalize the score *somehow*.

**Lucene.NET**

This is where the big question mark is. Here are my thoughts so far on Stack&amp;nbsp;Overflow functionality as well as how and what I&#39;ve already done.

**Indexing**

**Questions/Models** 

I believe each model should have an index of its own containing a unique id to quickly look it up based on a Term instance of that id (indexed, not analyzed).

In this area, I&#39;ve considered having Lucene.NET analyze each question/model and each reply individually.  So if there was one question and five answers, the question and each of the answers would be indexed as one unit separately.  

The idea here is that the relevance score that Lucene.NET returns would be easier to compare between models that project in different ways (say, something without replies).

As an example, a question sets the subject, and then the answer elaborates on the subject.

For a note, which doesn&#39;t have replies, it handles the matter of presenting the subject and then elaborating on it.

I believe that this will help with making the relevance scores more relevant to each other.

**Tags**

Initially, I thought that these should be kept in a separate index with multiple fields which have the ids to the documents in the appropriate model index. Or, if that&#39;s too large, there is an index with just the tags and another index which maintains the relationship between the tags index and the questions they are applied to. This way, when you click on a tag (or use the URL structure), it&#39;s easy to see in a progressive manner that you only have to &quot;buy into&quot; if you succeed:

- If the tag exists
- Which questions the tags are associated with
- The questions themselves

However, in practice, doing a query of all items based on tags (like clicking on a tag in Stack&amp;nbsp;Overflow) is *extremely easy* with SQL Server 2008. Based on the model above, it simply requires a query such as:

    select
         m.Name, m.Body
    from
        Models as m
            left outer join TagAssociations as ta on
                ta.ModelTypeId = &lt;fixed model type id&gt; and
                ta.ModelId = m.Id
            left outer join Tags as t on t.Id = ta.TagId
    where
        t.Name = &lt;tag&gt;

And since certain properties are shared across all models, it&#39;s easy enough to do a `UNION` between different model types/tables and produce a consistent set of results.

This would be analagous to a [`TermQuery`][10] in Lucene.NET (I&#39;m referencing the [Java documentation][11] since it&#39;s comprehensive, and Lucene.NET is meant to be a line-by-line translation of [Lucene][12], so all the documentation is the same).

The issue that comes up with using Lucene.NET here is that of sort order. The relevance score for a TermQuery when it comes to tags is irrelevant. It&#39;s either 1 or 0 (it either has it or it doesn&#39;t).

At this point, the confidence score (Wilson score interval) comes into play for ordering the results.

This score could be stored in Lucene.NET, but in order to sort the results on this field, it would rely on the values being stored in the field cache, which is something I really, really want to avoid. For a large number of documents, the field cache can grow very large (the Wilson score is a double, and you would need one double for every document, that can be one large array).

Given that I can change the SQL statement to order based on the Wilson score interval like this:

    select
         m.Name, m.Body
    from
        Models as m
            left outer join TagAssociations as ta on
                ta.ModelTypeId = &lt;fixed model type id&gt; and
                ta.ModelId = m.Id
            left outer join Tags as t on t.Id = ta.TagId
            left outer join VoteTallyStatistics as s on
                s.ModelTypeId = ta.ModelTypeId and
                s.ModelId = ta.ModelId
    where
        t.Name = &lt;tag&gt;
    order by
        --- Use Id to break ties.
        s.WilsonIntervalLowerBound desc, m.Id

It seems like an easy choice to use this to handle the piece of Stack&amp;nbsp;Overflow functionality &quot;get all items tagged with &amp;lt;tag&amp;gt;&quot;.

**Replies**

Originally, I thought this is in a separate index of its own, with a key back into the Questions index.

I think that there should be a combination of each model and each reply (if there is one) so that relevance scores across different models are more &quot;equal&quot; when compared to each other.

This would of course bloat the index.  I&#39;m somewhat comfortable with that right now.

Or, is there a way to store say, the models and replies as individual documents in Lucene.NET and then take both and be able to get the relevance score for a query treating *both documents as one*?  If so, then this would be **ideal**.

There is of course the question of what fields would be stored, indexed, analyzed (all operations can be separate operations, or mix-and-matched)? *Just how much would one index?*

What about using special stemmers/porters for spelling mistakes (using Metaphone) as well as synonyms (there is terminology in the community I will service which has it&#39;s own slang/terminology for certain things which has multiple representations)?

**Boost**

This is related to indexing of course, but I think it merits it&#39;s own section.

Are you **boosting** fields and/or documents? If so, how do you boost them? Is the boost constant for certain fields? Or is it recalculated for fields where vote/view/favorite/external data is applicable.

For example, in the document, does the title get a boost over the body? If so, what boost factors do you think work well? What about tags?

The thinking here is the same as it is along the lines of Stack&amp;nbsp;Overflow. Terms in the document have relevance, but if a document is tagged with the term, or it is in the title, then it should be boosted.

[Shashikant Kore][13] suggests a document structure like this:

- Title
- Question
- Accepted Answer (Or highly voted answer if there is no accepted answer)
- All answers combined

And then using boost but not based on the raw vote value. I believe I have that covered with the Wilson Score interval.

The question is, should the boost be applied to the entire document? I&#39;m leaning towards no on this one, because it would mean I&#39;d have to reindex the document each time a user voted on the model.

**Search for Items Tagged &lt;tag&gt;**

I originally thought that when querying for a tag (by specifically clicking on one or using the URL structure for looking up tagged content), that&#39;s a simple TermQuery against the tag index for the tag, then in the associations index (if necessary) then back to questions, Lucene.NET handles this really quickly.

However, given the notes above regarding how easy it is to do this in SQL Server, I&#39;ve opted for that route when it comes to searching tagged items.

**General Search**

So now, the most outstanding question is when doing a general phrase or term search against content, what and how do you integrate other information (such as votes) in order to determine the results in the proper order? For example, when performing this search on [ASP.NET MVC on Stack Overflow][14], these are the tallies for the top five results (when using the relevance tab):

        q votes answers accepted answer votes asp.net highlights mvc highlights
        ------- ------- --------------------- ------------------ --------------
             21      26                    51                  2              2
             58      23                    70                  2              5
             29      24                    40                  3              4
             37      15                    25                  1              2
             59      23                    47                  2              2

Note that the highlights are only in the title and abstract on the results page and are only minor indicators as to what the true term frequency is in the document, title, tag, reply (however they are applied, which is another good question).

How is all of this brought together?  

At this point, I know that Lucene.NET will return a normalized relevance score, and the vote data will give me a Wilson score interval which I can use to determine the confidence score.

**How should I look at combining tese two scores to indicate the sort order of the result set based on relevance and confidence?**

It is obvious to me that there should be *some* relationship between the two, but what that relationship should be evades me at this point.  I know I have to refine it as time goes on, but I&#39;m really lost on this part.

My initial thoughts are if the relevance score is beween 0 and 1 and the confidence score is between 0 and 1, then I could do something like this:

    1 / ((e ^ cs) * (e ^ rs))

This way, one gets a normalized value that approaches 0 the more relevant and confident the result is, and it can be sorted on that.

The main issue with that is that if boosting is performed on the tag and or title field, then the relevance score is outside the bounds of 0 to 1 (the upper end becomes unbounded then, and I don&#39;t know how to deal with that).

Also, I believe I will have to adjust the confidence score to account for vote tallies that are completely negative.  Since vote tallies that are completely negative result in a Wilson score interval with a lower bound of 0, something with -500 votes has the same confidence score as something with -1 vote, or 0 votes.

Fortunately, the upper bound decreases from 1 to 0 as negative vote tallies go up. I could change the confidence score to be a range from -1 to 1, like so:

    confidence score = votetally &lt; 0 ? 
        -(1 - wilson score interval upper bound) :
        wilson score interval lower bound

The problem with this is that plugging in 0 into the equation will rank all of the items with zero votes below those with negative vote tallies.

To that end, I&#39;m thinking if the confidence score is going to be used in a reciprocal equation like above (I&#39;m concerned about overflow obviously), then it needs to be reworked to always be positive.  One way of achieving this is:

    confidence score = 0.5 + 
        (votetally &lt; 0 ? 
            -(1 - wilson score interval upper bound) :
            wilson score interval lower bound) / 2

My other concerns are how to actually perform the calculation given Lucene.NET and SQL Server.  I&#39;m hesitant to put the confidence score in the Lucene index because it requires use of the field cache, which can have a huge impact on memory consumption (as mentioned before).

An idea I had was to get the relevance score from Lucene.NET and then using a [table-valued parameter][15] to stream the score to SQL Server (along with the ids of the items to select), at which point I&#39;d perform the calculation with the confidence score and then return the data properly ordred.

As stated before, there are a lot of other questions I have about this, and the answers have started to frame things, and will continue to expand upon things as the question and answers evovled.

  [1]: https://meta.stackexchange.com/questions/39950/how-is-lucene-net-utilized-on-stackoverflow-if-at-all
  [2]: http://incubator.apache.org/lucene.net/
  [3]: https://blog.stackoverflow.com/2008/11/sql-2008-full-text-search-problems/
  [4]: https://meta.stackoverflow.com/users/2/geoff-dalgas
  [5]: http://www.asp.net/mvc/
  [6]: http://msdn.microsoft.com/sqlserver
  [7]: http://msdn.microsoft.com/netframework
  [8]: http://msdn.microsoft.com/netframework
  [9]: http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval
  [10]: http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/TermQuery.html
  [11]: http://lucene.apache.org/java/3_0_1/
  [12]: http://lucene.apache.org/java/docs/
  [13]: https://stackoverflow.com/users/77308/shashikant-kore
  [14]: https://stackoverflow.com/search?q=asp.net+mvc
  [15]: http://msdn.microsoft.com/en-us/library/bb510489.aspx


How would one use Lucene.NET to help implement search on a site like Stack Overflow?

I&#39;m looking for a good *open source* (with LGPL or a permissive license) indexing engine for a node.js application, something like Lucene. I&#39;m looking for in-process indexing and search and am not interested in indexing servers like Sphinx or Solr.

I am not afraid to create bindings for a C/C++ library either so I&#39;m open to those kind of suggestions as well.


So far I&#39;ve found

* **node-clucene** which doesn&#39;t seem to be actively maintained anymore (and has several open issues)
* I could create my own binding for **CLucene** but it seems to be quite sparsely maintained and its current version is also quite behind the Java Lucene
* **Apache Lucy** which seems to be designed for the purpose of creating bindings for dynamic languages, but so far they don&#39;t have node bindings (nor a C API) and I haven&#39;t found any docs about creating bindings. I also didn&#39;t find any benchmarks about its performance.
* **node-search** which seems to be abandoned
* **jsii** which seems to be still a prototype and is also abandoned
* **fullproof** which is only intended to run in a web broswer
* **lunr.js** which seems to only allow serializing the whole index, so isn&#39;t scalable

I could &quot;roll my own&quot;, but I&#39;d prefer to use an already existing solution.

*EDIT: Why I&#39;m not interested in a standalone index server:* I use a fast in-process key-value store database, so it&#39;d be quite a waste having to go out of process for querying.

Is there a good indexing / search engine for Node.js?

I was under the assumption that each shard in Elasticsearch is an index. But I read somewhere that each segment is a Lucene index. 

What exactly is a segment? How does it effect search performance? I have indices that reach around 450GB in size everyday (I create a new one everyday) with default Elasticsearch settings.

When I execute `curl -XPOST &quot;http://localhost:9200/logstash-2013.03.0$i_optimize?max_num_segments=1&quot;`,  I get 
`num_committed_segments=11` and `num_search_segments=11`. 

Shouldn&#39;t the above values be 1? Maybe it&#39;s because of `index.merge.policy.segments_per_tier` value? What is this tier anyway?

Understanding Segments in Elasticsearch

I am running a Solr instance on Jetty and when I search using the Solr admin panel, it returns the entire document. What should I do to get only specified fields from each Solr document returned by the search?

Retrieving specific fields in a Solr query?

I have a cluster of 3 ElasticSearch nodes running on AWS EC2.  These nodes are setup using OpsWorks/Chef.  My intent is to design this cluster to be very resilient and elastic (nodes can come in and out when needed).

From everything I&#39;ve read about ElasticSearch, it seems like no one recommends putting a load balancer in front of the cluster; instead, it seems like the recommendation is to do one of two things:

1. Point your client at the URL/IP of one node, let ES do the load balancing for you and hope that node never goes down.

2. Hard-code the URLs/IPs of ALL your nodes into your client app and have the app handle the failover logic.

My background is mostly in web farms where it&#39;s just common sense to create a huge pool of autonomous web servers, throw an ELB in front of them and let the load balancer decide what nodes are alive or dead.  Why does ES not seem to support this same architecture?

Is using a load balancer with ElasticSearch unnecessary?


I struggle with understanding the difference between **collections and cores**. If I understand it correctly, cores are multiple indexes. Collection consists of cores, so essentially they share the same logic in separation, i.e. separate cores and collections have separate end-points.

I have the following scenario. I create a backend for cloud service for several online shops. Each shop has a set of products, to which customers can add reviews. I want to index static data (product information) separately from dynamic information(reviews) so I can improve performance.

How can I best separate in Solr???

Solr Collection vs Cores

I&#39;m following the advice given here in order to find partial words with elasticsearch:

https://stackoverflow.com/questions/5044674/elasticsearch-n-gram-tokenfilter-not-finding-partial-words

I&#39;ve created a simple bash script that attempts to run a version of this:

    curl -XDELETE 10.160.86.134:9200/products
    curl -XPOST 10.160.86.134:9200/products -d &#39;{
      &quot;index&quot;: {
        &quot;number_of_shards&quot;: 1,
        &quot;analysis&quot;: {
           &quot;filter&quot;: {
             &quot;mynGram&quot; : {&quot;type&quot;: &quot;nGram&quot;, &quot;min_gram&quot;: 2, &quot;max_gram&quot;: 10}
           },
           &quot;analyzer&quot;: {
             &quot;a1&quot; : {
               &quot;type&quot;:&quot;custom&quot;,
               &quot;tokenizer&quot;: &quot;standard&quot;,
               &quot;filter&quot;: [&quot;lowercase&quot;, &quot;mynGram&quot;]
             }
           }
         }
        }
      }
    }&#39;

    curl -XPUT 10.160.86.134:9200/products/_mapping -d &#39;{
      &quot;product&quot; : {
        &quot;index_analyzer&quot; : &quot;a1&quot;,
        &quot;search_analyzer&quot; : &quot;standard&quot;,
        &quot;properties&quot; : {
          &quot;product_description&quot;: {&quot;type&quot;:&quot;string&quot;},
          &quot;product_name&quot;: {&quot;type&quot;:&quot;string&quot;}
        }
      }
    }&#39;

Following running this script the first two commands (dumping products, then setting the index) seem to work giving me this:

    {&quot;ok&quot;:true,&quot;acknowledged&quot;:true}
    {&quot;ok&quot;:true,&quot;acknowledged&quot;:true}

Then it errors out following the mapping call giving me this:

    {&quot;error&quot;:&quot;ActionRequestValidationException[Validation Failed: 1: mapping type is missing;]&quot;,&quot;status&quot;:500}

Can anyone see what I&#39;m doing wrong?  Searching google starts autocompleting &quot;mapping not found elasticsearch&quot; so it seems to be a very common error.

Elasticsearch always returning &quot;mapping type is missing&quot;

I&#39;m looking for a way to do query auto-completion/suggestions in Lucene. I&#39;ve Googled around a bit and played around a bit, but all of the examples I&#39;ve seen seem to be setting up filters in Solr. We don&#39;t use Solr and aren&#39;t planning to move to using Solr in the near future, and Solr is obviously just wrapping around Lucene anyway, so I imagine there must be a way to do it!

I&#39;ve looked into using EdgeNGramFilter, and I realise that I&#39;d have to run the filter on the index fields and get the tokens out and then compare them against the inputted Query... I&#39;m just struggling to make the connection between the two into a bit of code, so help is much appreciated!

To be clear on what I&#39;m looking for (I realised I wasn&#39;t being overly clear, sorry) - I&#39;m looking for a solution where when searching for a term, it&#39;d return a list of suggested queries. When typing &#39;inter&#39; into the search field, it&#39;ll come back with a list of suggested queries, such as &#39;internet&#39;, &#39;international&#39;, etc.

All Lucene Solutions on Gang of Coders

Total of 26 Lucene Solutions