The entire ReactiveSearch stack is now completely free and open-source: API gateway, dashboard, search relevancy, pipelines, AI search, analytics, access control, caching, UI builder and more. No paywalls, no hosted-only features.

What is ReactiveSearch?

ReactiveSearch is an API gateway that sits between your client applications and Elasticsearch (or OpenSearch). It supercharges your search cluster with:

• Search Relevancy — fine-tune results without touching Elasticsearch configs

• Pipelines — composable, scriptable query and indexing pipelines

• AI Search — vector search, kNN, hybrid search out of the box

• Analytics — search analytics, click analytics, conversions

• Access Control — API keys, Role-based access, field-level security

• Speed & Caching — query caching and rate limiting

• UI Builder — no-code search UI builder

• Data Management — import, browse and manage your indices visually

It works with any client — React, Vue, Flutter, React Native, vanilla JavaScript, REST APIs — or the built-in no-code UI builder.

Get Started in 4 Steps

The entire stack runs with Docker. Here's how to get it up in under 5 minutes.

1. Clone and start the services

git clone https://github.com/appbaseio/reactivesearch-api-docker.git \
  && cd reactivesearch-api-docker

docker-compose -f docker-compose-with-elasticsearch.yaml up -d

This starts Elasticsearch, ReactiveSearch API, Nginx (with TLS) and Fluent Bit (for log shipping) — all with a single command.

docker-compose -f docker-compose-with-opensearch.yaml up -d

2. Verify the service is running

curl http://localhost:8000 -u rs-admin-user:rs-password

You should see a response like:

{
  "name": "elasticsearch",
  "cluster_name": "docker-cluster",
  "version": {
    "number": "9.3.0",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "17b451d8...",
    "build_date": "2026-01-29T10:05:46.708Z",
    "build_snapshot": false,
    "lucene_version": "10.3.2"
  },
  "tagline": "You Know, for Search"
}

This confirms ReactiveSearch is running and connected to your search cluster.

3. Connect the Dashboard

Open dash.reactivesearch.io in your browser and sign in.

4. Start building

After signing in you'll land on the Cluster Overview — your central hub for managing indices, configuring search relevancy, building search UIs, setting up analytics and more.

Why Open Source

Open source has always been central to ReactiveSearch. Projects like ReactiveSearch UI (1M+ installs, 4.9K GitHub stars) and Dejavu (4.7M Docker pulls, 8.5K GitHub stars) have been community staples for years.

And with ReactiveSearch project having reached a maturity, it now makes sense to give it fully to the community. With the current state of AI tooling, anyone can self-host, customize and extend it without friction. This is the most complete version of ReactiveSearch, and now it's yours.

The API and UI libraries are available under the Apache 2.0 license (and MIT license for some UI libraries).

You can support the project by sponsoring ReactiveSearch on GitHub.

Resources

• Quickstart Guide — Import data, preview search and build your first search UI

• Create a Search Pipeline — Learn how to create a search pipeline

• Build a Search UI — Hands-on demos with ReactiveSearch UI libraries

• GitHub — Star the repo, fork it, contribute

ReactiveSearch API enables you to build bespoke search UIs by taking search intent and transforming it (structured queries, AI embeddings, filters) directly into optimized search engine DSL queries. This lets you focus on designing relevance, not wrangling low-level DSL syntax.

Why Vector Search?

Vector search (kNN) opens up semantic and similarity-based use cases:

• Semantic Text Search: Retrieve documents based on their semantic similarity rather than keyword overlap.

• Image Similarity: Rank images by feature vectors (embeddings), enabling “find similar” functionality.

• Recommendation Engines: Find near-neighbor items for personalization or content-based recommendations.

• Anomaly Detection: Identify outliers by distance metrics in high-dimensional embedding space.

Many production teams already rely on Elasticsearch for text-based queries and can now easily incorporate vector-based similarity with the same cluster.

A Quick Demo of ReactiveSearch kNN

ReactiveSearch simplifies vector search by letting you combine it with typical search filters. Here’s an example request to the _reactivesearch API endpoint showing a kNN query combined with a term filter (tags.keyword) and a traditional text-based search (one_liner):

This request is translated into a native Elasticsearch DSL query using the knn clause and a filter. The vector_data field holds high-dimensional embeddings, and the final ranking of results is determined by vector similarity.

{
    "_source": {
        "excludes": [],
        "includes": [
            "name",
            "one_liner",
            "tags"
        ]
    },
    "knn": {
        "field": "vector_data",
        "filter": {
            "bool": {
                "must": [
                    {
                        "bool": {
                            "must": [
                                {
                                    "bool": {
                                        "filter": {
                                            "term": {
                                                "tags.keyword": "Fashion"
                                            }
                                        }
                                    }
                                },
                                {
                                    "bool": {
                                        "must": {
                                            "bool": {
                                                "minimum_should_match": 1,
                                                "should": [
                                                    {
                                                        "multi_match": {
                                                            "fields": [
                                                                "one_liner"
                                                            ],
                                                            "operator": "or",
                                                            "query": "dresses",
                                                            "type": "cross_fields"
                                                        }
                                                    },
                                                    {
                                                        "multi_match": {
                                                            "fields": [
                                                                "one_liner"
                                                            ],
                                                            "fuzziness": 0,
                                                            "operator": "or",
                                                            "query": "dresses",
                                                            "type": "best_fields"
                                                        }
                                                    },
                                                    {
                                                        "multi_match": {
                                                            "fields": [
                                                                "one_liner"
                                                            ],
                                                            "operator": "or",
                                                            "query": "dresses",
                                                            "type": "phrase"
                                                        }
                                                    },
                                                    {
                                                        "multi_match": {
                                                            "fields": [
                                                                "one_liner"
                                                            ],
                                                            "operator": "or",
                                                            "query": "dresses",
                                                            "type": "phrase_prefix"
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    }
                ]
            }
        },
        "k": 12,
        "num_candidates": 12,
        "query_vector": [
            -0.002622403,
            -0.008986259,
            -0.012236752,
            ....
            //1536 values
        ]
    },
    "size": 10
}

ReactiveSearch automatically translates this high-level request into a detailed Elasticsearch kNN query with integrated filters. By simply specifying your intent, you get advanced vector semantics wrapped in a developer-friendly interface.

Note: ReactiveSearch is able to translate the search text input into an embedding vector based on a pre-configured model that can set through AI preferences.

For embedding model choice, there is support for text-embedding-3-small, text-embedding-3-large and text-embedding-ada-002.

From Intent to Performance: Leveraging Elasticsearch at Scale

Since v8.16.0, Elasticsearch has been released as open source under the AGPL and has received multiple performance boosts in the 8.x series that make vector search production-ready at scale. Notable enhancements include improved concurrency for kNN graph traversal, advanced quantization (8‑bit/4‑bit) to reduce memory usage, and integrated filtering that enables sub‑50 ms kNN queries even when applying term or range constraints. Combined with robust indexing and search concurrency, these updates ensure that Elasticsearch can efficiently handle millions of vector documents.

Elasticsearch Performance & Tuning

• Mapping & Index Creation:

• Enable index:true on dense_vector fields for approximate kNN.

• Consider quantized field types (int8_hnsw or int4_hnsw) to drastically reduce memory usage.

• For large embeddings (up to 4096 dims), ensure your cluster runs Elasticsearch 8.11+.

• HNSW Parameters:

• m: Higher means better recall at the cost of more memory.

• ef_construction: Boost indexing quality vs. speed trade-off.

• num_candidates: Raise this at query time for higher accuracy (with some latency increase).

• Index Lifecycle & Segment Management:

• Avoid overly frequent refresh intervals to limit overhead when building HNSW graphs.

• Use force merges or well-sized shards to keep query latencies low.

• Filtering & Hybrid Queries:

• Elasticsearch 8.x supports filters directly in knn queries.

• Combine text-based matches and vector queries in the same DSL call for powerful hybrid search.

Ready to add vector search to your application?

With ReactiveSearch API v8.24.0, you can seamlessly turn semantic intent—like text queries, AI-driven embeddings, and domain filters—into optimized Elasticsearch kNN queries. From sub-50ms latencies on millions of documents to powerful hybrid filtering and re-ranking, Elasticsearch 8.x has become a production-ready powerhouse for vector-based retrieval.

Ready to get started? Just add a vectorDataField in your ReactiveSearch queries and fine-tune Elasticsearch’s HNSW parameters. You’ll be serving lightning-fast semantic search results in no time. Create a new search cluster or use the upgrade option from your cluster dashboard to use v8.24.0

In our previous post, we explored how vector (KNN) search lets you search for what you mean rather than what you type. In this post, we’re taking it one step further by integrating faceting capabilities into our search UI. By combining semantic search with faceting, users can filter results (for example, by industry) and get even more precise matches.

Note: With ReactiveSearch UI, you can now seamlessly integrate vector search capabilities with most of your UI components. This hybrid search approach merges vector search with traditional keyword search techniques to deliver contextual, precise results across all of your data in any modality, with less effort.

Try out the live demo: Knn search with faceting demo

2. Why Faceting Matters

• Refined Results: Facets let users quickly narrow down search outputs by specific categories.
• Enhanced Usability: Merging semantic search with traditional filters creates a more intuitive experience.
• Hybrid Search: This approach combines vector search for contextual understanding with faceting for categorical filtering.

Under the hood, ReactiveSearch leverages the Approximate kNN query DSL in Elasticsearch and OpenSearch, applying faceting during the kNN search phase to deliver precise, context-aware results.

3. Setting Up the Environment

Before diving in, ensure that you:

• Set up a ReactiveSearch cluster using ReactiveSearch Cloud (compatible with both Elasticsearch and OpenSearch).
• Index your vector dataset. You can use this indexing utility to help get started.
• Install the necessary dependencies:
  • @appbaseio/reactivesearch (for building the search UI)
  • @emotion/styled (for styling components)
  • react-icons (for adding iconography)

Note:

• Vector Field Mapping: Configure the field used for vector indexing and searching as a dense_vector in your Elasticsearch index mapping. If you're using the index utility provided by Elasticsearch, it will configure this for you.

4. Building the Faceted KNN Search UI

ReactiveBase and SearchBox

We start by wrapping our UI with ReactiveBase to connect to the search cluster and add a SearchBox for capturing semantic queries.

<ReactiveBase
  app="yc-companies-dataset"
  url="https://appbase-demo-ansible-abxiydt-arc.searchbase.io"
  credentials="a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61"
  transformRequest={(request) => {
    // Custom query transformation logic
  }}
>
  <Container>
    <h2>K-Nearest Neighbors Search with Facets</h2>
    <SearchBox
      componentId="search"
      dataField={["name", "one_liner"]}
      placeholder="Semantic search for startup companies"
      autosuggest={false}
      style={{ marginBottom: "1rem" }}
      URLParams
    />
    {/* Additional layout components go here */}
  </Container>
</ReactiveBase>

Faceting with MultiList

Next, add a MultiList component that enables filtering by industries. It is configured to react to the search input.

<FacetContainer>
  <MultiList
    componentId="industries"
    dataField="industries.keyword"
    title="Industries"
    placeholder="Filter by industries"
    showSearch={false}
    react={{ and: ["search"] }}
    style={{ marginBottom: "1rem" }}
  />
</FacetContainer>

Displaying Results with ReactiveList

The ReactiveList component renders our search results. Each result displays company details like logo, name, description, team size, stage, industry tags, and a website link.

<ResultsContainer>
  <ReactiveList
    componentId="results"
    vectorDataField="vector_data"
    dataField="_score"
    size={20}
    pagination={false}
    react={{ and: ["search", "industries"] }}
    includeFields={[
      "name",
      "one_liner",
      "long_description",
      "team_size",
      "stage",
      "industries",
      "website",
      "small_logo_thumb_url",
    ]}
    render={({ data }) => (
      <>
        {data.map((item) => {
          const company = item._source || item;
          return (
            <ResultItem key={company._id}>
              <Logo
                src={company.small_logo_thumb_url}
                alt={`${company.name} logo`}
              />
              <Info>
                <CompanyName>{company.name}</CompanyName>
                <OneLiner>
                  {company.one_liner || company.long_description}
                </OneLiner>
                <Meta>
                  <span>
                    <FaUsers /> {company.team_size}
                  </span>
                  <span>
                    <FaBuilding /> {company.stage}
                  </span>
                </Meta>
                <Tags>
                  {company.industries &&
                    company.industries.map((ind) => (
                      <Tag key={ind}>{ind}</Tag>
                    ))}
                </Tags>
                <Link
                  href={company.website}
                  target="_blank"
                  rel="noopener noreferrer"
                >
                  Visit website
                </Link>
              </Info>
            </ResultItem>
          );
        })}
      </>
    )}
    renderNoResults={() => <div>No results found</div>}
  />
</ResultsContainer>

Customizing Search Queries

The transformRequest function customizes the search query before sending it to Elasticsearch. It dynamically adjusts the payload based on whether a semantic query or industry filter (or both) are applied.

transformRequest={(request) => {
  const body = JSON.parse(request.body);
  const searchQuery = body.query.find((q) => q.id === "search");
  const industriesQuery = body.query.find((q) => q.id === "industries");
  const resultsQueryIndex = body.query.findIndex((q) => q.id === "results");
  const resultsQuery = body.query.find((q) => q.id === "results");

  if (
    (searchQuery?.value && searchQuery.value !== "") ||
    (industriesQuery?.value && industriesQuery.value.length > 0)
  ) {
    if (resultsQuery) {
      resultsQuery.candidates = resultsQuery.size;
      if (searchQuery.value) {
        resultsQuery.value = searchQuery.value;
        resultsQuery.react = { and: ["industries"] };
      } else if (industriesQuery.value) {
        resultsQuery.value = industriesQuery.value.join(" ");
      }
      if (resultsQuery.dataField) {
        delete resultsQuery.dataField;
      }
      body.query[resultsQueryIndex] = resultsQuery;
    }
  } else {
    body.query.splice(resultsQueryIndex, 1);
  }
  return {
    ...request,
    body: JSON.stringify(body),
  };
}}

5. Code Walkthrough

• ReactiveBase: Connects the UI to your ReactiveSearch/Elasticsearch backend.
• SearchBox: Captures semantic queries from users.
• MultiList: Implements faceting by filtering search results based on industry.
• ReactiveList: Displays the results in a structured and styled list.
• transformRequest: Adjusts the search query dynamically based on active filters.

Each component works together to create a hybrid search experience that combines vector search with traditional faceting.

6. Live Demo and Final Thoughts

Check out the live demo here: KNN with Facets Demo. You can browse the code for the demo over here: https://github.com/awesome-reactivesearch/opensource-demos/blob/main/src/pages/KNN/knn-with-faceting.jsx.

In this follow-up guide, we enhanced our original vector search UI by incorporating faceting. This approach allows users to refine search results effectively, making the overall search experience more precise and user-friendly. Leveraging the ReactiveSearch UI kit, you can easily combine the power of hybrid search techniques with traditional filters to deliver an unmatched search experience. Stay tuned for more advanced search UI enhancements using ReactiveSearch!

Internally, a vector search engine stores a numerical representation of each data: be it text, images, or other types of media even, called embeddings. These embeddings capture the underlying meaning and context of the data rather than just the exact words or pixels. When you search using vector search, the engine compares the embedding of your query to the embeddings of all stored items. It finds the results that are most semantically similar, even if they don’t contain the exact words or elements you used in your search.

This means you can search with vague descriptions, synonyms, or even related concepts, and still find the right match because vector search understands what you’re looking for, not just the exact terms you use. This makes it an ideal choice for product discovery, recommendation systems, natural-language and voice search use-cases.

The Ingredients

In this post, I will show how to build a vector search UI using ReactiveSearch for the search UI, OpenSearch as the vector search engine, and OpenAI embeddings API for creating the numerical vector representation.

ReactiveSearch offers an API for authoring search endpoints as well as an open-source UI kit for building search UIs with React, Vue, and Flutter.

OpenSearch is a versatile open-source keyword and vector search engine. The latest OpenSearch 2.17.0 release introduces several vector search enhancements,

The vector search UI allows searching from over 5,000 Y-Combinator companies. You can try out the live demo at: https://oss-demos.reactivesearch.io/knn.

Create Search Cluster

We start by creating a search cluster with ReactiveSearch Cloud at https://reactivesearch.io. This will create a deployment of the OpenSearch engine and the ReactiveSearch API service.

If you already have an OpenSearch cluster hosted, you can deploy the ReactiveSearch API service using one of these docker-compose files.

Indexing Data

We start by indexing the dataset of over 5,000 Y-Combinator companies into the ReactiveSearch (with OpenSearch engine configured) cluster.

If you already have an OpenSearch index, you can use the KNN re-index utility https://github.com/appbaseio/ai-scripts/tree/master/knn_reindex to generate embeddings based on your existing data + mappings and re-index the data. It uses the text-embedding-3-small model from OpenAI for embedding, this can be updated by forking the utility.

Authoring Search Pipeline

Next, I will author a search pipeline with the ReactiveSearch dashboard. A ReactiveSearch pipeline allows stitching the search engine (here, OpenSearch) with additional connectors to enrich search queries, add transformation steps, or enrich search engine results. The pipeline definition is based on a JSON format and these steps can be written in JavaScript. Check out these how-to guides to see what you can build with pipelines.

{
    "enabled": true,
    "description": "KNN query pipeline",
    "routes": [
        {
            "path": "/_knn_search/_reactivesearch",
            "method": "POST",
            "classify": {
                "category": "reactivesearch"
            }
        }
    ],
    "envs": {
        "index": "yc-companies-dataset"
    },
    "stages": [
        {
            "id": "authorize user",
            "use": "authorization"
        },
        {
            "id": "fetch embeddings",
            "use": "openAIEmbeddings",
            "inputs": {
                "apiKey": "{{openAIConfig.open_ai_key}}",
                "useWithReactiveSearchQuery": true
            },
            "continueOnError": false
        },
        {
            "id": "os_query_body",
            "scriptRef": "os_query",
            "continueOnError": false
        },
        {
            "use": "httpRequest",
            "description": "HTTP query to OS",
            "inputs": {
                "url": "http://localhost:9200/{{index}}/_search",
                "body": "{{{osBody}}}",
                "headers": {
                    "Content-Type": "application/json"
                },
                "method": "POST"
            }
        }
    ]
}

Here, we first get the OpenAI embedding for the incoming search query, then generate the query DSL body using the os_query script (below) and then send this to the OpenSearch engine to return search results.

function handleRequest() {
    // Parse the request body
    console.log('request body is: ', context.request.body);
    reqBody = JSON.parse(context.request.body);

    // Construct the OpenSearch DSL query
    var osBody = JSON.stringify({
        query: {
            knn: {
                vector_data: {
                    vector: reqBody.query[0].queryVector,
                    k: 10,
                },
            },
        },
        _source: [
            'name',
            'one_liner',
            'website',
            'small_logo_thumb_url',
            'team_size',
            'industries',
            'regions',
            'stage',
        ],
        size: 10,
    });

    // Return the updated context with the new esBody
    return { ...context, osBody };
}

The ReactiveSearch dashboard provides an execution environment where I can set the REST API request body and see the live response from the pipeline. It also provides console logs and individual stage changes that can be helpful while developing.

We just configured the pipeline to respond to this API endpoint:

curl -XPOST https://search-server.com/_knn_search/reactivesearch -d '{
    "query": [
        {
            "id": "search",
            "value": "Semantic search API",
            "vectorDataField": "vector_data"
        }
    ]
}'

The pipeline fetches the OpenAI embeddings first for the search input, and then creates the query DSL to run a KNN search query against the OpenSearch engine. The results are then returned back without any modifications.

Building The Search UI

Now that the REST API is configured, the next and final step is to build a search UI. Here, I’m choosing to build a basic Search UI with a ChatGPT prompt.

### Task

Write a React component named `KNN` that implements a K-Nearest Neighbors (KNN) based search UI.

### Server Connection Info

- **POST method**: `https://appbase-demo-ansible-abxiydt-arc.searchbase.io/_knn_search/_reactivesearch`
- **API Payload**:

{
  "query": [
    {
      "id": "search",
      "value": "Semantic search API",
      "vectorDataField": "vector_data"
    }
  ]
}

Response is returned in Elasticsearch hits format.

### Component Requirements

React with styled-components based on @emotion

### Search Functionality

- Implement an input field for search queries.
- Include a button labeled **"Try random search input"** to autofill a random query from a predefined list.
- Use **Basic Auth** with credentials:
  - Username: `a03a1cb71321`
  - Password: `75b6603d-9456-4a5a-af6b-a487b309eb61`

### State Management

- Use `useState` to manage:
  - `searchInput`
  - `results`
  - `loading`
  - `noResults`
  - `searchTime`
- Use `useEffect` and `useRef` to handle side effects and input debouncing.

### Result Display

- Display search results in a styled list with:
  - **Company logo**  `small_logo_thum_url` field
  - **Company name**  `name`
  - **One-liner description** `one_liner`
  - **Team size and stage** (with icons from `react-icons/fa`) `team_size`, `stage`
  - **Industries** as tags `industries`
  - **Link** to the company’s website. `website`

### Loading and No Results States

- Show a loading message during search.
- Display a message if no results are found.

Please provide the complete code.

This should generate a close to functional code. Here is a version with some tweaks to the ChatGPT output: KNN/index.jsx.

You can try out the live demo at: https://oss-demos.reactivesearch.io/knn.

Summary

In this guide, we explored how to build a vector search UI using ReactiveSearch and OpenSearch 2.17. I started by creating a search cluster with ReactiveSearch Cloud, then indexed a dataset of over 5,000 Y-Combinator companies. Next, I authored a search pipeline to handle the search queries and fetch results using OpenAI embeddings. Finally, I built a simple search UI to interact with the search API.

In the next post, we will see how to build a hybrid search with KNN-search along with faceting, where we will use the ReactiveSearch UI kit for building the search UI.

Internally, a vector search engine stores a numerical representation of each data: be it text, images, or other types of media even, called embeddings. These embeddings capture the underlying meaning and context of the data rather than just the exact words or pixels. When you search using vector search, the engine compares the embedding of your query to the embeddings of all stored items. It finds the results that are most semantically similar, even if they don’t contain the exact words or elements you used in your search.

This means you can search with vague descriptions, synonyms, or even related concepts, and still find the right match because vector search understands what you’re looking for, not just the exact terms you use. This makes it an ideal choice for product discovery, recommendation systems, natural-language and voice search use-cases.

The Ingredients

In this post, I will show how to build a vector search UI using ReactiveSearch for the search UI, Elasticsearch as the vector search engine, and OpenAI embeddings API for creating the numerical vector representation.

ReactiveSearch offers an API for authoring search endpoints as well as an open-source UI kit for building search UIs with React, Vue, and Flutter.

Elasticsearch is a versatile keyword and vector search engine. Particularly of note, in the last year, Elasticsearch has significantly enhanced its vector search capabilities, with improvements in indexing and querying vector fields from versions 7.x to 8.15, including support for approximate nearest neighbor (ANN) search and optimizations for high-dimensional data.

The vector search UI allows searching from over 5,000 Y-Combinator companies. You can try out the live demo at: https://oss-demos.reactivesearch.io/knn.

Create Search Cluster

We start by creating a search cluster with ReactiveSearch Cloud at https://reactivesearch.io. This will create a deployment of the Elasticsearch engine and the ReactiveSearch API service.

If you already have an Elasticsearch cluster hosted, either as OpenSearch or with Elasticsearch Cloud, you can deploy the ReactiveSearch API service using one of these docker-compose files.

Indexing Data

We start by indexing the dataset of over 5,000 Y-Combinator companies into the ReactiveSearch (with Elasticsearch engine configured) cluster.

If you already have an Elasticsearch index, you can use the KNN re-index utility https://github.com/appbaseio/ai-scripts/tree/master/knn_reindex to generate embeddings based on your existing data + mappings and re-index the data. It uses the text-embedding-3-small model from OpenAI for embedding, this can be updated by forking the utility.

Authoring Search Pipeline

Next, I will author a search pipeline with the ReactiveSearch dashboard. A ReactiveSearch pipeline allows stitching the search engine (here, Elasticsearch) with additional connectors to enrich search queries, add transformation steps, or enrich search engine results. The pipeline definition is based on a JSON format and these steps can be written in JavaScript. Check out these how-to guides to see what you can build with pipelines.

{
    "enabled": true,
    "description": "KNN query pipeline",
    "routes": [
        {
            "path": "/_knn_search/_reactivesearch",
            "method": "POST",
            "classify": {
                "category": "reactivesearch"
            }
        }
    ],
    "envs": {
        "index": "yc-companies-dataset"
    },
    "stages": [
        {
            "id": "authorize user",
            "use": "authorization"
        },
        {
            "id": "fetch embeddings",
            "use": "openAIEmbeddings",
            "inputs": {
                "apiKey": "{{openAIConfig.open_ai_key}}",
                "useWithReactiveSearchQuery": true
            },
            "continueOnError": false
        },
        {
            "id": "es_query_body",
            "scriptRef": "es_query",
            "continueOnError": false
        },
        {
            "use": "httpRequest",
            "description": "HTTP query to ES",
            "inputs": {
                "url": "http://localhost:9200/{{index}}/_search",
                "body": "{{{esBody}}}",
                "headers": {
                    "Content-Type": "application/json"
                },
                "method": "POST"
            }
        }
    ]
}

Here, we first get the OpenAI embedding for the incoming search query, then generate the query DSL body using the es_query script (below) and then send this to the Elasticsearch engine to return search results.

// es_query.js
function handleRequest() {
    reqBody = JSON.parse(context.request.body);
    var esBody = JSON.stringify({
        knn: {
            field: 'vector_data',
            query_vector: reqBody.query[0].queryVector,
            k: 10,
            num_candidates: 100,
        },
        _source: [
            'name',
            'one_liner',
            'website',
            'small_logo_thumb_url',
            'team_size',
            'industries',
            'regions',
            'stage',
        ],
    });
    // sets esBody at the top-level in the context,
    return { ...context, esBody };
}

The ReactiveSearch dashboard provides an execution environment where I can set the REST API request body and see the live response from the pipeline. It also provides console logs and individual stage changes that can be helpful while developing.

We just configured the pipeline to respond to this API endpoint:

curl -XPOST https://search-server.com/_knn_search/reactivesearch -d '{
    "query": [
        {
            "id": "search",
            "value": "Semantic search API",
            "vectorDataField": "vector_data"
        }
    ]
}'

The pipeline fetches the OpenAI embeddings first for the search input, and then creates the query DSL to run a KNN search query against the Elasticsearch engine. The results are then returned back without any modifications.

Building The Search UI

Now that the REST API is configured, the next and final step is to build a search UI. Here, I’m choosing to build a basic Search UI with a ChatGPT prompt.

### Task

Write a React component named `KNN` that implements a K-Nearest Neighbors (KNN) based search UI.

### Server Connection Info

- **POST method**: `https://appbase-demo-ansible-abxiydt-arc.searchbase.io/_knn_search/_reactivesearch`
- **API Payload**:

{
  "query": [
    {
      "id": "search",
      "value": "Semantic search API",
      "vectorDataField": "vector_data"
    }
  ]
}

Response is returned in Elasticsearch hits format.

### Component Requirements

React with styled-components based on @emotion

### Search Functionality

- Implement an input field for search queries.
- Include a button labeled **"Try random search input"** to autofill a random query from a predefined list.
- Use **Basic Auth** with credentials:
  - Username: `a03a1cb71321`
  - Password: `75b6603d-9456-4a5a-af6b-a487b309eb61`

### State Management

- Use `useState` to manage:
  - `searchInput`
  - `results`
  - `loading`
  - `noResults`
  - `searchTime`
- Use `useEffect` and `useRef` to handle side effects and input debouncing.

### Result Display

- Display search results in a styled list with:
  - **Company logo**  `small_logo_thum_url` field
  - **Company name**  `name`
  - **One-liner description** `one_liner`
  - **Team size and stage** (with icons from `react-icons/fa`) `team_size`, `stage`
  - **Industries** as tags `industries`
  - **Link** to the company’s website. `website`

### Loading and No Results States

- Show a loading message during search.
- Display a message if no results are found.

Please provide the complete code.

This should generate a close to functional code. Here is a version with some tweaks to the ChatGPT output: KNN/index.jsx.

You can try out the live demo at: https://oss-demos.reactivesearch.io/knn.

Summary

In this guide, we explored how to build a vector search UI using ReactiveSearch and Elasticsearch 8.15. I started by creating a search cluster with ReactiveSearch Cloud, then indexed a dataset of over 5,000 Y-Combinator companies. Next, I authored a search pipeline to handle the search queries and fetch results using OpenAI embeddings. Finally, I built a simple search UI to interact with the search API.

In the next post, we will see how to build a hybrid search with KNN-search along with faceting, where we will use the ReactiveSearch UI kit for building the search UI.

Search functionality is a vital component of modern applications, significantly enhancing user experience by providing quick access to relevant information. As Flutter continues to gain popularity for building cross-platform apps, integrating a robust search interface has become increasingly important. Enter Flutter Searchbox 4.0, a package that offers a declarative API to seamlessly connect your Flutter app with Elasticsearch or OpenSearch, enabling the creation of powerful search UIs.

In this post, we’ll guide you through setting up a basic search interface using Flutter Searchbox 4.0, highlighting its key features and providing a practical example.

Understanding Flutter Searchbox 4.0

Flutter Searchbox is designed to integrate seamlessly with ReactiveSearch, making it easier to query Elasticsearch and OpenSearch. It allows developers to build responsive and dynamic search UIs with minimal code, leveraging pre-built widgets for autosuggestions, results rendering, facets and maps.

Installation

Before diving into coding, let’s set up the necessary dependencies.

1. Depend on it
   Add the following dependencies to your pubspec.yaml file:

    codedependencies:
      flutter_searchbox: ^4.0.0
      searchbase: ^4.0.0
   

2. Install it
   Run the following command to install the packages:

    flutter pub get
   

Basic Usage

Now that we have the dependencies installed, let’s build a simple search UI.

A Simple Example

The following example demonstrates how to create a search interface with an autosuggestion search widget and a custom result widget.

import 'package:flutter/material.dart';
import 'package:searchbase/searchbase.dart';
import 'package:flutter_searchbox/flutter_searchbox.dart';

void main() {
  runApp(FlutterSearchBoxApp());
}

class FlutterSearchBoxApp extends StatelessWidget {
  final searchbaseInstance = SearchBase(
    'good-books-ds',
    'https://appbase-demo-ansible-abxiydt-arc.searchbase.io',
    'a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61',
    appbaseConfig: AppbaseSettings(
      recordAnalytics: true,
      userId: 'test@dev',
    ),
  );

  @override
  Widget build(BuildContext context) {
    return SearchBaseProvider(
      searchbase: searchbaseInstance,
      child: MaterialApp(
        title: "SearchBox Demo",
        theme: ThemeData(
          primarySwatch: Colors.blue,
          visualDensity: VisualDensity.adaptivePlatformDensity,
        ),
        home: HomePage(),
      ),
    );
  }
}

class HomePage extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('SearchBox Demo'),
        actions: [
          IconButton(
            icon: Icon(Icons.search),
            onPressed: () {
              showSearch(
                context: context,
                delegate: SearchBox(
                  id: 'search-widget',
                  enableRecentSearches: true,
                  enablePopularSuggestions: true,
                  showAutoFill: true,
                  maxPopularSuggestions: 3,
                  size: 10,
                  dataField: [
                    {'field': 'original_title', 'weight': 1},
                    {'field': 'original_title.search', 'weight': 3},
                  ],
                ),
              );
            },
          ),
        ],
      ),
      body: Center(
        child: SearchWidgetConnector(
          id: 'result-widget',
          dataField: 'original_title',
          react: {
            'and': ['search-widget'],
          },
          size: 10,
          triggerQueryOnInit: true,
          preserveResults: true,
          builder: (context, searchController) => ResultsWidget(searchController),
        ),
      ),
    );
  }
}

class ResultsWidget extends StatelessWidget {
  final SearchController searchController;
  ResultsWidget(this.searchController);

  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        Card(
          child: Align(
            alignment: Alignment.centerLeft,
            child: Container(
              color: Colors.white,
              height: 20,
              child: Text(
                '${searchController.results.numberOfResults} results found in ${searchController.results.time.toString()} ms',
              ),
            ),
          ),
        ),
        Expanded(
          child: ListView.builder(
            itemCount: searchController.results.data.length + 1,
            itemBuilder: (context, index) {
              WidgetsBinding.instance.addPostFrameCallback((_) {
                var offset = (searchController.from ?? 0) + searchController.size;
                if (index == offset - 1 && searchController.results.numberOfResults > offset) {
                  searchController.setFrom(offset, options: Options(triggerDefaultQuery: true));
                }
              });

              return index < searchController.results.data.length
                ? Container(
                    margin: const EdgeInsets.all(0.5),
                    padding: const EdgeInsets.fromLTRB(0, 15, 0, 0),
                    decoration: BoxDecoration(border: Border.all(color: Colors.black26)),
                    height: 200,
                    child: Row(
                      children: [
                        Expanded(
                          flex: 3,
                          child: Card(
                            semanticContainer: true,
                            clipBehavior: Clip.antiAliasWithSaveLayer,
                            child: Image.network(
                              searchController.results.data[index]["image_medium"],
                              fit: BoxFit.fill,
                            ),
                            elevation: 5,
                            margin: EdgeInsets.all(10),
                          ),
                        ),
                        Expanded(
                          flex: 7,
                          child: Column(
                            children: [
                              ListTile(
                                title: Text(
                                  searchController.results.data[index]["original_title"],
                                  style: TextStyle(fontSize: 20.0),
                                ),
                                subtitle: Text(
                                  'By: ${searchController.results.data[index]["authors"]}',
                                  style: TextStyle(fontSize: 15.0),
                                ),
                              ),
                              Row(
                                children: [
                                  Padding(
                                    padding: const EdgeInsets.fromLTRB(10, 5, 0, 0),
                                    child: Text(
                                      '(${searchController.results.data[index]["average_rating"]} avg)',
                                      style: TextStyle(fontSize: 12.0),
                                    ),
                                  ),
                                ],
                              ),
                              Row(
                                children: [
                                  Padding(
                                    padding: const EdgeInsets.fromLTRB(27, 10, 0, 0),
                                    child: Text(
                                      'Pub: ${searchController.results.data[index]["original_publication_year"]}',
                                      style: TextStyle(fontSize: 12.0),
                                    ),
                                  )
                                ],
                              ),
                            ],
                          ),
                        ),
                      ],
                    ),
                  )
                : searchController.requestPending
                  ? Center(child: CircularProgressIndicator())
                  : ListTile(
                      title: Center(
                        child: RichText(
                          text: TextSpan(
                            text: searchController.results.data.length > 0
                              ? "No more results"
                              : 'No results found',
                            style: TextStyle(color: Colors.black54, fontSize: 20, fontWeight: FontWeight.bold),
                          ),
                        ),
                      ),
                    );
            },
          ),
        ),
      ],
    );
  }
}

This code sets up a basic search interface where users can search for books, get autosuggestions, and view detailed results, including images, titles, authors, and publication years.

Customizing the Search Experience

Flutter Searchbox offers flexibility in customizing search experiences. You can tailor search widgets, adjust search parameters, and integrate additional features like filters and facets to refine results based on user preferences.

Optimizing Performance and User Experience

Optimizing your search interface is crucial for delivering a fast and responsive user experience. Consider implementing caching strategies, managing large datasets efficiently, and ensuring the UI remains smooth even under heavy search loads.

Summary

Building search UIs with Flutter Searchbox 4.0 is efficient and customizable, offering a rich set of features to create a seamless search experience. By following the steps in this guide, you can add powerful search capabilities into your Flutter app.

For more examples with facets, custom Search UI, and maps, explore the Flutter Searchbox documentation and start building search-powered apps today.

Watch the 1-min introduction video below: 👇

New Components

AI Answer

The AIAnswer UI component enables building a RAG (retrieval augmented generation) based chat experience with AI answering support. See a demo in action over here. Or read the component's documentation reference over here.

SearchBox

The SearchBox component is enhanced to support AI Answering, image searching, showing FAQs as suggestions, and showing curated suggestions with custom actions. See all of these in action with an extensive search showcase below. Read the component documentation over here.

ReactiveCharts

Embed reactive charts in your search interface. With 5 built-in chart types: pie, bar, scatter, line and histogram supported and use ReactiveChart to support any chart UI using Apache E-charts.

Simplified SSR

With ReactiveSearch v4, configuring SSR is an intuitive experience by removing the requirement of providing the component configuration.

Secure API

ReactiveSearch v4 uses the ReactiveSearch API introduced in v3 to express the search intent. This requires the use of a ReactiveSearch server with two initial deployment choices: a search cluster and a serverless search mode.

Expanded Search Engine Support

ReactiveSearch UI kit supports Elasticsearch, OpenSearch, OpenAI, Solr and MongoDB Atlas Search as the choices for backend search engines.

Elasticsearch and OpenSearch are supported completely. OpenAI is supported with both ReactiveSearch API and with pipelines. Solr and MongoDB Atlas search supports are in preview, we're happy to provide additional support to get you up and running with these engines.

I'm excited to see what you build with ReactiveSearch v4! You can check out the developer quickstart guide and star the project at https://github.com/appbaseio/reactivesearch.

You can check out this post for the ReactiveSearch Vue v3 release.

Understanding ReactiveSearch

With native connectors for Elasticsearch, OpenSearch, Solr and OpenAI, ReactiveSearch enables businesses to choose an incremental search improvement and adoption strategy over betting on a completely new search solution.

How it works

Connect search engine index, LLM such as OpenAI's ChatGPT, or integrate a HTTP endpoint as a search data source.

Then, author a pipeline that takes a HTTP request as input, modifies the request by either using ReactiveSearch's pre-built stages or by writing JavaScript, queries search sources via connectors, and then transforms, re-orders and optionally enriches the response which is then served to the client. This orchestration happens in milliseconds, our benchmarks mark a typical pipeline doing the above adding a ~10ms overhead and can scale up to 300 QPS (that's about ~1MM requests in an hour). V8 engine isolates make a significant contribution as the choice of environment for executing JavaScript ⚡️ fast.

As the above image illustrates, the orchestration of the search pipeline is written as a YAML (or JSON) and the stages each represent a middleware. Here, the authorization, reactiveSearchQuery and elasticsearchQuery are pre-built stages and the user is writing a JavaScript for google_knowledge_graph and merge_response stages, to make a fetch request to Google's Knowledge Graph API and to merge the search engine and KG responses before serving it to the client. Stages that are I/O heavy and independent can be executed asynchronously as is the case here for the KG API call and the Elasticsearch Query stages.

Pipelines come with starter templates for over 10 use cases today and the authoring interface is via Monaco editor (what powers VS Code) with autocompletion and usage tooltip support to help you with the syntax. The JavaScript syntax supported by V8 is very close to the browser JavaScript runtimes -- we will illuminate the usage of all of this with a use-case of building an E-Commerce search pipeline.

Crafting a Context-Aware E-Commerce Search

A context-aware E-Commerce search should:

• Display Recent and Popular suggestions when no query is entered.
• Provide relevant suggestions during active user input.
• Offer AI-generated answers to user questions.
• Handle typos during product searches.

To visualize the end result, refer to the embedded demo below.

Search Index

Before we build the search, we will need to index the data. For this use case, we've ingested 100,000 products from different categories through the Best Buy developer API. You likely already have a search index configured in Elasticsearch, OpenSearch or Solr, so we will skip over this part.

We will start with the ReactiveSearch dashboard and take a closer look at both the search index and the Browse Data functionality.

At this point, you probably want to get a free 14-day evaluation ↗️ of the ReactiveSearch platform. Choose Search Cluster for a green field project, or choose Serverless Search if you're connecting to an existing search index.

Building the Search Pipeline

Let's take a closer look at the process of building a search pipeline. We will start out with a pre-built pipeline template, orchestrate the pipeline stages with JSON and then write some JavaScript for building the context-aware search query.

Since we intend to use the no-code UI builder for building the search UI in the following step, we will assume the request payload to be using the ReactiveSearch API - a declarative API for capturing search intent. Pipelines by themselves make no assumptions of the use of ReactiveSearch API, though it certainly helps with being able to use the pre-built stages as well as the no-code UI builder.

The orchestration for the pipeline that we defined is as follows:

{
    "enabled": true,
    "description": "Best Buy Search Pipeline",
    "routes": [
        {
            "path": "/best-buy-set-pipeline/_reactivesearch",
            "method": "POST",
            "classify": {
                "category": "reactivesearch"
            }
        }
    ],
    "envs": {
        "index": [
            "best-buy-set-2023"
        ]
    },
    "stages": [
        {
            "id": "auth",
            "use": "authorization"
        },
        {
            "id": "generateRequest",
            "scriptRef": "generateRequest",
            "continueOnError": true
        },
        {
            "id": "query",
            "use": "reactivesearchQuery",
            "continueOnError": false
        },
        {
            "id": "es_query",
            "use": "elasticsearchQuery",
            "continueOnError": false
        },
        {
            "id": "typo check",
            "scriptRef": "checkTypo",
            "continueOnError": false
        },
        {
            "id": "researchQuery",
            "use": "reactivesearchQuery",
            "continueOnError": false,
            "trigger": {
                "expression": "context.envs.research == true"
            }
        },
        {
            "id": "research_es_query",
            "use": "elasticsearchQuery",
            "continueOnError": false,
            "trigger": {
                "expression": "context.envs.research == true"
            }
        },
        {
            "id": "answerAI",
            "use": "AIAnswer",
            "inputs": {
                "topDocsForContext": 3,
                "docTemplate": "${source.name}",
                "queryTemplate": "Can you tell me about: ${value}",
                "apiKey": "{{ OPENAI_API_KEY }}"
            }
        }
    ]
}

Our first custom script takes the browser query and modifies it to set the search and suggestion queries to use based on the search intent, which is detected based on the presence/absence of user input as well as the specificity of it.

The second custom script for applying typo tolerance is triggered when no hits are returned by the original query.

Below is a template for how a JavaScript stage should be defined.

// your function handler should always be named as handleRequest()
function handleRequest() {
    // Accessible variables within the function: context,
    // e.g. 1. context.envs contains the envs set by the pipeline and dynamically at runtime
    //      2. JSON.parse(context.request.body) provides the JSON of the request body - useful for changes to request body
    //      3. JSON.parse(context.response.body) provides the JSON of the request body - useful for enriching the response body
    // The function expects a return value of the context if you're making changes to the request or response body or setting a variable
    // e.g. return { ...context, myVar: myVar }
    console.log('request body is: ', context.request.body);
    var myVar = 'test';
    // sets myVar at the top-level in the context,
    return { ...context, myVar };
}

As this template shows, a custom JavaScript stage will define a function handleRequest() { handler that may read the request context, modify it, set a new variable in the context or add console logs.

Read the pipeline docs for concepts and how-to guides.

Building the search UI

Now that we have the search pipeline configured, let's consume it with a search UI. For this use case, we will choose to do this with the no-code UI builder. ReactiveSearch UI kit is also a good choice for building a React, Vue or Flutter based search UI.

Summary

ReactiveSearch offers a versatile stack for incrementally adopting AI to an existing business search offering an alternative to betting on new solutions. It does this by providing connectors to Elasticsearch, OpenSearch, Solr and OpenAI. In this post, we went over setting up the search index, browsing the data, building the search pipeline for context-aware search, and finally building the search UI.

Browse other interactive use-cases of ReactiveSearch over here ↗️

Get a free 14-day evaluation ↗️ of the ReactiveSearch platform

We've curated the top 10 applications of SearchBox along with interactive consumer-grade search demos in React and Vue.

1. Voice Search

This example demonstrates how to enable voice search by setting the showVoiceSearch prop to true. This adds a microphone icon to the search box, allowing users to search using their voice instead of typing. This can be especially useful for users on mobile devices or for those with accessibility needs.

React code + demo for Voice Search

Vue code + demo for Voice Search

2. Auto suggestions

This example demonstrates how to enable auto-suggestions. When the autosuggest prop is set to true, the SearchBox will query the search index for suggestions based on the user's input.

React code + demo for Auto suggestions

Vue code + demo for Auto suggestions

3. Auto Suggestions + Popular + Recent Suggestions

This example shows how to enable popular and recent suggestions along with auto-suggestions based on a search index. Both popular and recent suggestions are powered by ReactiveSearch analytics captured from end-users and offer search time configurations for setting index, and minimum thresholds for characters, hits and # of times a query is searched.

React code + demo

Vue code + demo

4. Instant Search

This example demonstrates how to enable instant search by setting the autosuggest prop to false. With instant search, users can instantly see the effect of searching at each keystroke on the facets and results.

React code + demo for Instant Search

Vue code + demo for Instant Search

5. Instant Search + Suggestions As Pills

There are times when you want to get the best of both worlds: Display results instantly along with showing possible autocomplete extensions to the user. This example uses controlled behavior with a custom rendering of suggestions to achieve this.

React code + demo

Vue code + demo

6. Keyboard Shortcut to Focus

Configure familiar keyboard shortcuts and keyboard accessibility for your SaaS power users.

React code + demo for keyboard shortcuts

Vue code + demo for keyboard shortcuts

7. Featured Suggestions

Featured (aka curated) suggestions that show up as default suggestions, and provide for rich interactive behavior such as custom styling, navigate on click and more. One can set featured suggestions using the point-and-click interface of the ReactiveSearch dashboard and then display them alongside auto-suggestions, and popular or recent suggestions.

React code + demo for Featured Suggestions

Vue code + demo for Featured Suggestions

8. Question Answering with AIAnswer

This is a simple AI question answering UI that uses the AIAnswer component along with SearchBox and ReactiveList.

React code + demo for AIAnswer

Vue code + demo for AIAnswer

9. Question Answering with follow-up questions

Same same as question answering with AIAnswer component but also allow your users to ask follow-up questions. AI Answer component supports custom rendering allowing you as a developer to provide a rich user experience while leveraging ReactiveSearch's query structure.

React code + demo for question answering with follow-up questions support

Vue code + demo for question answering with follow-up questions support

The SearchBox component auto-adjusts its height to up to three rows to allow for a question.

10. AI Answering integrated into SearchBox

SearchBox as a Swiss army knife where your users get an answer along with suggestions. Props allow configuring how this can work seamlessly.

React code + example for SearchBox showing an AI Answer

Vue code + demo for SearchBox showing an AI Answer

Summary

In this post, we just showcased the top 10 applications of SearchBox UI to create an engaging user experience.

SearchBox and ReactiveSearch UIs are open-source (Apache 2.0 licensed) and designed to work with a ReactiveSearch cloud instance.

ReactiveSearch cloud is a reverse proxy for your favorite search engines and provides a no-code control plane to author search pipelines (connect your favorite search engines, use pre-built stages or add JavaScript processing stages), build and ship UIs and get insights into your search.

Sign up for a free 14-day trial of ReactiveSearch cloud

Stay tuned for more updates and features in the ReactiveSearch UI library. Happy shipping!

In this article, I explore how ChatGPT can be used for documentation and knowledge base search in user and customer support scenarios, allowing users to find answers to their questions in a conversational way. However, it's important to note that ChatGPT has its limitations, especially regarding its knowledge cutoff. It can't access data beyond 2021, so any changes or updates to your documentation after this date won't be reflected in its responses.

This limitation means that certain queries related to your documentation may result in outdated or incorrect answers, as ChatGPT doesn't have up-to-date information beyond its cutoff date of September 2021. In this article, I'll show you how to overcome this limitation of ChatGPT by integrating context from a search engine to ensure more accurate responses.

Here's the application I'm going to build:

LTM - QA

ChatGPT works as a GQA (Generative Question Answering) system. A basic GQA system only requires a user text query and a large language model (LLM).

I enhance GQA with LTM-QA, which stands for Long-Term Memory Question Answering. LTM-QA allows machines to store information for a long time and recall it when needed. This improves both precision and recall - a win-win situation.

The innovative idea here is that I use BM-25 (TF/IDF) relevance instead of requiring data vectorization, although this is also possible - a post on the same is forthcoming. Elasticsearch and OpenSearch (the Apache 2.0 derivative of the former) are the go-to search engines when it comes to relevant text search with filtering capabilities. If your search is used by users who are familiar with the data, then BM-25-based relevance offers instant results at scale and query remixing capabilities that are very nascent with vector search today.

The key idea is to combine the power of ChatGPT and state-of-the-art search engines such as ElasticSearch and OpenSearch to create a robust search experience that's relevant, accurate, concise, and evolves as per the users' prompts.

🤔 Elasticsearch and OpenSearch are the standards (10x adoption, work at scale, and have good Dx) where developers already have their long-term search data, and they support BM-25 and vector search-based relevance models.

ReactiveSearch is a CloudFlare for Search Engines

ReactiveSearch.io offers a supercharged search experience for creating the most demanding app search experiences with a no-code UI builder and search relevance control plane, AI Answering support (which I will be using here), pipelines to create an extendible search backend, caching, and search insights.

There are additional open-source and hosted tools that simplify the process of building search experiences. You can:

• Import your data from various sources via the dashboard or CLI or REST APIs,

• Test search relevancy visually,

• Build production-grade search UIs using:

  1. UI component libraries that are available for React, Vue, React Native, Flutter, and vanilla JavaScript,

  2. A declarative REST API or

  3. NoCode search UI builder.

• Get out-of-the-box actionable analytics on top searches, no-result searches, slow queries, and more,

• Get improved search performance and throughput with application layer caching,

• Build access-controlled search experiences with built-in Basic Auth or JWT-based authentication, read/write access keys with granular ACLs, field level security, IP-based rate limits, and time to live - read more over here.

With recent upgrades, ReactiveSearch supports all major search engine backends like OpenSearch, Mongo Atlas, Solr, OpenAI

Visit the website here.

Data Prep

As Elasticsearch works with JSON natively, I will create a search index using the JSON data of ReactiveSearch's documentation pages.

A document looks as below:

{
    meta_title: "ReactiveComponent",
    meta_description: "ReactiveComponent lets you connect any React UI component with an Elasticsearch query or an aggregation seamlessly.",
    keywords: ["react", "web", "reactivesearch", "ui"],
    heading: "Usage",
    tokens: "ReactiveComponent lets you connect any React UI component with an Elasticsearch query or an aggregation seamlessly. It can be used as a standalone component to integrate Elasticsearch queries into your frontend UI declaratively. It can also be used in conjunction with other ReactiveSearch components. Read more here."
}

The indexing structure I've decided to use here treats each section of a page as a separate document. The following image shows the fields mapped to the documentation page.

For your dataset, you can index using the ReactiveSearch dashboard's importer interface (which supports JSON, JSONL, and CSV formats) or with the REST API.

Building the Search UI

All the code I'm going to share is located inside the ask-reactivesearch repo. If you're new to it, I recommend going through the quick start guide of the ReactiveSearch UI library. I'll also introduce a new component called AIAnswer which creates an experience similar to ChatGPT.

Configuring ReactiveBase

// src/App.jsx
function Main() {
  return (
    <ReactiveBase
      app="reactivesearch_docs_v2"
      url="https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io"
      theme={{
        typography: {
          fontFamily: "monospace",
          fontSize: "16px",
        },
      }}
      reactivesearchAPIConfig={{
        recordAnalytics: true
      }}
    >
      {/*All components go here*/}
    </ReactiveBase>
  );
}

const App = () => <Main />;

export default App;

You can read about the props we're setting in more detail in the documentation reference over here.

Adding SearchBox Component

function Main() {
  return (
    <ReactiveBase {...config}>
      <SearchBox
        componentId="search"
        dataField={[
          {
            field: "keywords",
            weight: 4,
          },
          {
            field: "heading",
            weight: 1,
          },
          {
            field: "tokens",
            weight: 1,
          },
          {
            field: "meta_title",
            weight: 2,
          },
        ]}
        distinctField="meta_title.keyword"
        highlight={false}
        size={5}
        autosuggest={true}
        URLParams
        showClear
      />
    </ReactiveBase>
  );
}

I'm going to create a SearchBox component with a specific configuration. I've set up the dataField property to prioritize keywords, heading, tokens and meta_title. I figured out the best weights for these fields through a bit of trial and error.

I also want to show some FAQs, so I'll create custom suggestions. You can check out the code for the render function on Github and see the result in the Codesandbox below.

Rendering AI Response

There's one last step I want to share with you. It's about choosing to get an AI response for your search query right inside the SearchBox component. All I have to do is set enableAI to true, and just like that, I have a working AI searchbox. Now, since I'm creating suggestions myself, I also choose to render the AI response. The response and other properties come as parameters of my render function.

Adding an AI config is completely up to you. If you're interested in fine-tuning the precision of your AI response, it can be quite handy. You can find more details in the documentation.

<SearchBox
  enableAI
  AIConfig={{
    docTemplate:
      "title is '${source.title}', page content is '${source.tokens}', URL is https://docs.reactivesearch.io${source.url}",
    queryTemplate:
      "Answer the query: '${value}', cite URL in your answer from the context",
    topDocsForContext: 3,
  }}
  AIUIConfig={{
    askButton: true,
    renderAskButton: (clickHandler) => {
      return (
        <button className="ask-ai-button" onClick={clickHandler}>
          <span className="button-emoji">
            <ButtonSvg />
          </span>
          <span className="button-text">{"Ask AI"}</span>
        </button>
      );
    },
  }}
  render={({
          downshiftProps: {
            isOpen,
            getItemProps,
            highlightedIndex,
            selectedItem,
          },
          AIData: { answer: aiAnswer, isAILoading },
          data,
          loading,
        }) => {
    /* Rendering SearchBox with JSX */
  }}
/>;

The final app appears as follows:

Try searching for your ReactiveSearch questions and share your experience.

Summary

This blog post discusses the integration of ChatGPT and Elasticsearch for fact-based question-answering for documentation. It begins by introducing ChatGPT, a language model developed by OpenAI, and its application in answering questions and providing information on a wide range of topics, including technical documentation. The post then delves into the limitations of ChatGPT, particularly its inability to access data beyond 2021, and proposes a solution to this issue by integrating it with Elasticsearch, a search engine that provides context for more accurate responses.

The post provides a detailed walkthrough of building an application that leverages ChatGPT and Elasticsearch. It introduces LTM-QA (Long-Term Memory Question Answering), a system that enhances GQA (Generative Question Answering) by storing information for extended periods and recalling it as needed. The integration of BM-25 (TF/IDF) relevance with Elasticsearch and OpenSearch is also discussed.

The source code for the application is open-source, and you can adapt it to build your own AI answering search UI with ReactiveSearch.

Learn how to build an AI Answer interface for your own domain with this follow-along post - where we use ElasticSearch and ChatGPT together to build an AI Answering system with high precision (no hallucinations), is always based on facts from your domain (instead of outdated facts it may have) and leverages Elasticsearch as a search engine effectively to filter and show results along-side the AI Answer.

To keep the user interface configurable for your use-case, we will use the ReactiveSearch UI kit's (Apache 2.0 licensed) AIAnswer component here. This is available for both React and Vue today.

LTM - QA

ChatGPT is a GQA (Generative Question Answering) system. The most straightforward GQA system requires nothing more than a user text query and a large language model (LLM).

We enhance GQA with LTM-QA, which stands for Long-Term Memory Question Answering. LTM-QA enables machines to store information for an extended period and recall it as needed. This improves both precision and recall - so a win-win.

The novel idea here is that we make use of BM-25 (TF/IDF) relevance instead of requiring vectorization of data, although this is also possible - a post on the same is coming soon. Elasticsearch and OpenSearch (which is the Apache 2.0 derivative of the former) are the search engines to beat when it comes to relevant text search with filtering capabilities. If your search is used by users who know the data well, then BM-25 based relevance offers instant results at scale and query remixing capabilities that are very nascent with vector search today.

The simple idea is to combine the power of ChatGPT and standard search engine tools like ElasticSearch and OpenSearch to finally come up with a powerful search experience that's relevant, accurate, to the point, and evolving as per the users' prompts.

🤔 Elasticsearch and OpenSearch are the standards (10x adoption, work at scale, and have good Dx) where developers already likely have their long-term search data, also they both support BM-25 and vector search.

ReactiveSearch is the CloudFlare for Search Engines

ReactiveSearch.io provides a supercharged search experience for creating the most demanding app search experiences with a no-code UI builder and search relevance control plane, AI Answering support (what we will be using over here), pipelines to create an extendible search backend, caching and search insights.

There are additional open-source and hosted tools that simplify the process of building search experiences. You can:

• Import your data from various sources via the dashboard or CLI or REST APIs,

• Test search relevancy visually,

• Build production-grade search UIs using:

  1. UI component libraries that are available for React, Vue, React Native, Flutter, and vanilla JavaScript,

  2. A declarative REST API or

  3. NoCode search UI builder.

• Get out-of-the-box actionable analytics on top searches, no-result searches, slow queries, and more,

• Get improved search performance and throughput with application layer caching,

• Build access-controlled search experiences with built-in Basic Auth or JWT-based authentication, read/write access keys with granular ACLs, field level security, IP-based rate limits, and time to live - read more over here.

Dataset and relevant questions

To make a great UI we should have a good dataset. We would be using a dataset of dialogues from the popular TV Show "Rick and Morty". We would also need an index to store the data.

You can set up and install an Elasticsearch server by following the official installation guide, or you can create a free account at reactivesearch.io which provides Elasticsearch hosting as a service and is easy to use. For simplicity, we will be using reactivesearch.io service to get started.

I’ve already created an index with the dataset. You can check out the dataset from above over here in the data browser tool Dejavu, which is built by reactivesearch.io.

We also have a long list of questions in JSON format with the following structure. Each question is linked to an episode, which is useful for finding documents in the above index for a particular episode.

[
  {
    "question": "How many Ricks and Mortys are there?",
    "episode": "A Rickle in Time"
  },
  {
    "question": "What's the most bizarre time paradox Rick and Morty encounter?",
    "episode": "A Rickle in Time"
  },
  {
    "question": "What's the funniest time-related mishap?",
    "episode": "A Rickle in Time"
  },
]

You can also have a separate index for the questions if you have a large list which makes the search slow.

Building Search UI

All the code we are going to show is present inside the qna-rick-and-morty repository. We also recommend going through the quick start guide of the Reactivesearch UI library to become familiar with it. The guide will not only show you how to set up a boilerplate React app, but also how to set up the @appbaseio/reactivesearch library. We will primarily modify the src/App.jsx file, which will also introduce you to ReactiveBase and SearchBox component that we will be using in this tutorial. Additionally, we will introduce a new component called AIAnswer, which creates a ChatGPT-like search experience.

Configuring ReactiveBase:

We would want to connect to rick-and-morty index. For that we configure ReactiveBase component as follows:

// src/App.jsx
function Main() {
  return (
    <ReactiveBase
      app="reactivesearch_docs_v2"
      url="https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io"
      theme={{
        typography: {
          fontFamily: "monospace",
          fontSize: "16px",
        },
      }}
      reactivesearchAPIConfig={{
        recordAnalytics: false,
        userId: "jon",
      }}
    >
      {/*All components go here*/}
    </ReactiveBase>
  );
}

const App = () => <Main />;

export default App;

Adding SearchBox:

We would make a SearchBox component with the following config. You would be familiar with the props from the quick start guide. All the props are also listed in the documentation.

function Main() {
  return (
    <ReactiveBase
      {...config}
    >
      <SearchBox
        dataField={["episode_name", "name", "line"]}
        componentId={SEARCH_COMPONENT_ID}
        size={5}
        autosuggest={true}
      />
    </ReactiveBase>
  );
}

To search through the list of questions and show suggestions when the user types, we will be using a library called fuse.js. Fuse.js is a lightweight fuzzy search library that is appropriate for the size of our data. If the data were larger, we would have created an index for better performance.

You need to install it first. The latest version at the time of writing is v6.6.2.

yarn add fuse.js

You can look at the documentation for Fuse.js though the usage is quite simple. You just have to pass data to Fuse instance and then call the search method passing your searchQuery. Note that we append the episode name in the question itself which would be used to get the documents filtered by the episode.

// src/App.js
// questions: [{question, episode}]
const fuse = new Fuse(
  questions.map((q) => ({
    ...q,
    question: q.episode
      ? `${q.question.replace(/\?$/, "")} in "${q.episode}"?`
      : q.question,
  })),
  { keys: ["question"] }
);
const filteredData = fuse.search(searchQuery).map((res) => ({
  value: res.item.question,
  episode: res.item.episode,
  idx: res.refIndex,
}));

We would then render this filteredData using the custom render.

function App() {
  return (
    <ReactiveBase {...configProps}>
      <SearchBox
        {...previousProps}
        render={({
          downshiftProps: {
            isOpen,
            getItemProps,
            highlightedIndex,
            selectedItem,
          },
          data,
          value: searchQuery,
        }) => {
          const fuse = new Fuse(
            questions.map((q) => ({
              ...q,
              question: q.episode
                ? `${q.question.replace(/\?$/, "")} in "${q.episode}"?`
                : q.question,
            })),
            { keys: ["question"] }
          );
          const filteredData = fuse.search(searchQuery).map((res) => ({
            value: res.item.question,
            episode: res.item.episode,
            idx: res.refIndex,
          }));

          return isOpen ? (
            <div className={`${styles.suggestions}`}>
              <div>
                {filteredData.length ? (
                  <p className={`bg-gray p-2 m-0 ${styles.suggestionHeading}`}>
                    Try below suggestions
                  </p>
                ) : null}
                {filteredData.length ? (
                  <div>
                    {filteredData.map((item, index) =>
                      index < 6 ? (
                        <div
                          /* eslint-disable-next-line react/no-array-index-key */
                          key={item.idx}
                          {...getItemProps({
                            item,
                          })}
                          className={`${
                            highlightedIndex === index
                              ? styles.activeSuggestion
                              : styles.suggestion
                          } 
                                  ${
                                    selectedItem &&
                                    selectedItem.value === item.value
                                      ? styles.selectedSuggestion
                                      : ""
                                  }
                                  `}
                        >
                          <span className="clipText">{item.value}</span>
                        </div>
                      ) : null
                    )}
                  </div>
                ) : null}
              </div>
            </div>
          ) : null;
        }}
      />
    </ReactiveBase>
  );
}

Adding the AIAnswer Component

AIAnswer is an AI-driven answer UI component that interacts with a dataset to provide context-aware and relevant answers based on user inputs. It employs machine learning to comprehend user questions and retrieve the most pertinent information. The component can be used to supply answers to common questions related to a specific topic, offer support and help by addressing user queries, and create a knowledge base or FAQ section for your website.

Learn more about the AIAnswer component over here.

The AIAnswer component adheres to the same principles as other ReactiveSearch components. It has a react prop that lists dependencies. We will add the SearchBox component as a dependency so that when we hit Enter and the SearchBox triggers a query, the AIAnswer component will display the relevant answer.

There is an additional prop called AIConfig, which allows configuring the default prompt and parameters sent to ChatGPT. Referring to the diagram in the "Let's Visualize" section, we feed documents from Elasticsearch to ChatGPT. These documents are the top results found by performing a search (the suggestions seen in the SearchBox component). The format is determined by docTemplate prop, where we can specify any field from the search index (rick-and-morty in this case, this would vary based on your index). topDocsForContext dictates how many documents to feed to ChatGPT. queryTemplate is the format for the actual question fed to ChatGPT after the document queries, which is formed from the value provided in the SearchBox.

// src/App.js
function App() {
  return (
    <ReactiveBase
      {...configProps}
    >
      <SearchBox
        componentId={SEARCH_COMPONENT_ID}
        {...otherProps}
      />
      <div id="answer-component" className={`px-5 pt-5 ${styles.answer}`}>
        <AIAnswer
          componentId={AI_ANSWER_COMPONENT_ID}
          showVoiceInput
          showIcon
          react={{ and: SEARCH_COMPONENT_ID }}
          AIConfig={{
            docTemplate: "'${source.name}', says '${source.line}'",
            queryTemplate: "${value}",
            topDocsForContext: 15,
          }}
          enterButton={true}
          showInput={false}
        />
      </div>
    </ReactiveBase>
  );
}

In our case, we also want to show the documents that were used to generate the AI answer in the UI itself, so we know the answers come from the search index and the AI isn't hallucinating or replying from out-of-domain facts. We store the data in the state using the onData handler.

function App() {
  const [feedData, setFeedData] = useState({ loading: true, hits: [] });
  return (
    <ReactiveBase
      {...configProps}
    >
      <SearchBox
        componentId={SEARCH_COMPONENT_ID}
        {...otherProps}
      />
      <div id="answer-component" className={`px-5 pt-5 ${styles.answer}`}>
        <AIAnswer
          componentId={AI_ANSWER_COMPONENT_ID}
          onData={({ loading, rawData, data }) => {
            if (!loading && data.length) {
              if (rawData.hits && rawData.hits.hits)
                setFeedData({
                  loading: false,
                  hits: rawData.hits.hits.map((hit) => hit._source),
                });
            } else {
              setFeedData({ loading: true, hits: [] });
            }
          }}
          {...otherProps}
        />
      </div>
    </ReactiveBase>
  );
}

Getting episode from the search query

Since we store the associated episode for each question suggestion, we would like to use this information in getting the first-pass results from the search engine. In a previous step, we put the episode name in the question, and now we will extract it and tell the index to only show those documents which match the episode name.

We will use the transformRequest method to do this, which lets us modify the payload generated by the ReactiveSearch library before sending it across the network.

function App() {
  const [feedData, setFeedData] = useState({ loading: true, hits: [] });
  return (
    <ReactiveBase
      {...configProps}
      transformRequest={(req) => {
        const body = JSON.parse(req.body);
        body.query = body.query.map((componentQuery) => {
          if (
            componentQuery.id === SEARCH_COMPONENT_ID &&
            componentQuery.type === "search"
          ) {
            const searchQuery = componentQuery.value;
            const matches = searchQuery.match(/in "(.*?)"\?$/);
            const episode = matches && matches[1];

            return {
              ...componentQuery,
              customQuery: {
                query: {
                  term: {
                    "episode_name.keyword": episode,
                  },
                },
              },
            };
          }
          return componentQuery;
        });
        body.settings = {
          recordAnalytics: false,
          backend: "opensearch",
        };
        const newReq = { ...req, body: JSON.stringify(body) };
        return newReq;
      }}
    >
    </ReactiveBase>
  );
}

The complete app should look as below:

Summary

Hope you enjoyed reading! Here's a brief recap of what we covered, so you can implement the same for a domain of your choice:

• ChatGPT is a GQA system with limitations, which can be overcome by using LTM-QA to improve both precision and recall.

• We used a search index of Rick and Morty dialogues to retrieve documents from when user performs a search.

• We utilized the SearchBox component to suggest a list of popular questions that a user might have.

• We added an AIAnswer component that depended on the SearchBox and updated dynamically based on the user's query.

• Lastly, we filtered documents relevant to the query (specific episode) to improve the precision of the answer.

1. Improve search results:

   KNN Search can be used to improve traditional search results by providing more relevant and accurate results to users. This can be particularly useful when searching for information whose meaning is difficult to describe in words. For example, if a user searches "Not good dog food", then traditional search might show results containing "good dog food" which is opposite to the user's intention.

2. Recommend items:
   It can be used to recommend items to users based on their preferences or behavior. For example, a recommendation system might use similarity search to find items that are similar to those that a user has previously purchased or viewed.

3. Categorizing data into clusters:

   KNN search can group similar items together, allowing us to identify patterns and relationships in the data. For example, an e-commerce company might use clustering to group customers based on their purchase behavior. Customers who frequently purchase items in a specific category or price range can be grouped together in a cluster, while customers who tend to make one-time purchases can be grouped in a different cluster. The company can then create targeted marketing campaigns for each cluster, such as offering discounts or promotions on items that are popular within a particular cluster.

Visualizing data as a vector space

Before we implement KNN Search we need to understand a fundamental concept on which it is built, "vectors". Vectors are simply arrays. While in programming we can have arrays of strings, objects, etc., vectors can contain only numeric values like [2, 3]. Hence vectors are numeric arrays. The cool part about vectors is that we can plot them on a graph and find distance between them and see how close one is to another.

We can't plot text or audio information on graph or can we? What if we transform the textual data to a vector. That is essentially what a machine learning model does. It takes real world objects like text, audio, etc. and generates a vector. A collection of vectors is called a vector space. You can see the space of sentences below:

Above is a 2 dimensional graph. Adding more dimensions to the vector means nearest points get more similar. A real world model can map data to thousands of dimensions.

What are we building?

We are going to build a search UI for a dataset of Amazon reviews of various products. This search wouldn't be a traditional search, but would show a user similar results. For example, if a user searches "Not good dog food", then traditional search might show results containing "good dog food" which might not be the intention. But KNN search would show similar results like "bad dog food" because it maps the two queries as nearest neighbours.

The Building Blocks

OpenAI Vector Embeddings API: Model for vectorizing data

We would use this API to convert our textual review data into vectors. You would need an API key to access the Embeddings API. You can do that by going to the OpenAI website and signing up.

OpenSearch: Search index

We use a search index that has the capability of performing a KNN search on vectors. For this, we will use an OpenSearch index because it doesn't have limitations as opposed to other search indexes.

💡 Note: ElasticSearch has a vector dimension limitation. This means the vector array in ElasticSearch can have a maximum length of 1024 but OpenAI vectors need an array of 1536 length. Read more about this limitation in ElasticSearch on this open issue.

Update: The issue looks addressed now and as Elasticsearch releases a new version addressing this limitation, we will be publishing a tutorial on the same using Elasticsearch.

Reactivesearch: The Catalyst

We have all the tools but we need the infrastructure to make the above pieces work with each other without you having to do any work. Reactivesearch.io provides hosting an OpenSearch index but you can also "Bring Your Own Cluster"(BYOC). We would also use a feature called pipelines which helps in organizing various steps like vectorizing data, indexing it, etc. into stages. This makes the process of developing such an application highly efficient. Reactivesearch also has a UI library which we would use to build the app.

Let's dive into code

We are going to use Reactivesearch.io pipelines to build this app. If you are unfamiliar with it, then you can read it in the documentation. In short, they help us to perform operations on data in various stages. The results can then be sent through as response to the front end.

Indexing pipeline:

Creating and configuring index:

First, we would need to create an index with vectorized data of our product reviews. We would do so by using an indexing pipeline. First things first, we would need to create an index named amazon_reviews. The index also needs to be created such that it is aware of the vectorized form of the data which is later going to be used for KNN search. We specify the field type as knn_vector and also have additional settings. Also, note that the field name is vector_data which would be referenced later when we build the frontend app. You can look at below curl script which you can run from the terminal with reactivesearch_cloud_url of your cluster. We would also need the index.

curl --location --request PUT 'https://{{reactivesearch_cloud_url}}/amazon_reviews' \
--header 'Content-Type: application/json' \
--data-raw '{
    "settings": {
        "knn": true,
        "knn.algo_param.ef_search": 100
    },
    "mappings": {
        "properties": {
            "vector_data": {
                "type": "knn_vector",
                "dimension": 1536,
                "method": {
                    "name": "hnsw",
                    "space_type": "cosinesimil",
                    "engine": "nmslib"
                }
            }
        }
    }
}'

Anatomy of pipeline:

We need a pipeline to first convert the raw product data into vector form. Then, we can take that data and index as knn_vector in search index. This would make us ready to perform KNN search.

Here is what our pipeline looks like in a config file. If you don't know how to setup an indexing pipeline using Reactivesearch.io you can follow the documentation. You can paste the below in the pipeline editor and it would convert itself to JSON config. You would need to paste OpenAI api key in envs field to make the pipeline work.

We want to vectorize to text data fields Text and Summary and we specify that in the stage.

Notice that we mention the output_key as vector_data. This is the field where the text information would be stored as vectors.

enabled: true
description: Index pipeline to store vectorized data
routes:
  - path: /amazon_reviews/_doc
    method: POST
    classify:
      category: elasticsearch
      acl: index

envs:
  openAIApiKey: ${{ OPENAI_API_KEY }}

stages:
- id: authorize user
  use: authorization
- id: fetch embeddings
  use: openAIEmbeddingsIndex
  inputs:
    apiKey: "{{openAIApiKey}}"
    inputKeys:
    - Summary
    - Text
    outputKey: vector_data
  continueOnError: false
- id: index data
  use: elasticsearchQuery
  needs:
  - fetch embeddings

Indexing data:

Once the pipeline is deployed at the above endpoint we can index the data. We have created a script to index data. You can follow the steps after creating the above indexing pipeline.

Search pipeline:

We just indexed our textual data (product reviews) inside the index. But when the user performs a search it is still text and not a vector. Hence we can't perform a KNN search without vectorizing the query value. Also, we shouldn't index the search query using the index pipeline because the search query isn't a product review and would be different each time. Instead, we create a new pipeline which transforms the search query using the OpenAI embedding into a vector and then gives it to the search index to perform KNN search.

Config of the pipeline looks like below.

enabled: true
routes:
- path: "/amazon_reviews/_reactivesearch"
  method: POST
  classify:
    category: reactivesearch

envs:
  openAIApiKey: ${{ OPENAI_API_KEY }}

stages:
- id: authorize user
  use: authorization
- id: fetch embeddings
  use: openAIEmbeddings
  inputs:
    apiKey: "{{openAIApiKey}}"
    useWithReactiveSearchQuery: true
  continueOnError: false
- use: reactivesearchQuery
  needs:
  - fetch embeddings
  continueOnError: false
- use: elasticsearchQuery
  continueOnError: false

Below shows a diagram of what might happen in the search phase. We already have the reviews inside the index(Green dots). We get the search query from the user and vectorize it and provide it to the search index, which maps it near to some reviews in the vector space (review 1 and 2). Those are the nearest neighbours and the results sent back to the user would reflect it i.e. Review 1 and 2 would be ranked higher, and would appear before other reviews. Note that we don't index the search query vector. It is temporary and would not be present in the next search.

Building the UI:

All the code we are going to show is present inside knn-search-demo github repo. We also would like you to go through the quick start guide of the Reactivesearch UI library to get some familiarity. The guide will not only show how to set up a boilerplate react app but also how to setup @appbaseio/reactivesearch library. We would primarily modify the src/App.jsx file and would be using ReactiveBase, SearchBox, and ReactiveList which the quick start guide already covers.

Connecting to the search pipeline:

In order to send the search query to the pipeline and getting results back we would need to establish a connection with it. We do it by using the endpoint property available on ReactiveBase. You would be able to get the credentials from dash.reactivesearch.io. Also, note that you would have to set reactivesearchAPIConfig.recordAnalytics to false.

import React from "react";
import {
  ReactiveBase,
} from "@appbaseio/reactivesearch";

function Main() {
  const HOST_URL = "https://{{user}}:{{password}}@{{host}}/amazon_reviews/_reactivesearch"

  return (
    <ReactiveBase
      endpoint={{
        url: HOST_URL,
        method: "POST",
      }}
      reactivesearchAPIConfig={{
        recordAnalytics: false,
        userId: "jon",
      }}
    >
      {/* Search and ReactiveList component go here */}
    </ReactiveBase>
  );
}

const App = () => <Main />;

export default App;

Adding search and result component:

In our dataset, there are two fields in which we are mainly interested, Summary and Text. Summary is a short description of the whole review(Text).

We would keep the dataField as Summary inside the Searchbox. This would be used in showing suggestions matching the search query (this would not be KNN search, rather a regular search). We would also control the value of the SearchBox component by using value and onChange props. This would be used later when we transform the normal search request. We also use debounce which fetches suggestions only after an interval when the user has stopped typing.

// src/App.jsx
function Main() {
  const [searchValue, setSearchValue] = useState("");

  return (
    <ReactiveBase
      {...configProps}
    >
      <SearchBox
        dataField={["Summary"]}
        componentId="SearchComponent"
        size={5}
        showClear
        value={searchValue}
        debounce={SUGGESTION_DEBOUNCE_DELAY}
        onChange={(value) => {
          setSearchValue(value);
        }}
      />
    </ReactiveBase>
  );
}

Next, we would add a component to show the results, ReactiveList. Make sure to set the react property to the componentId value of the SearchBox component, i.e. "SearchComponent". We can then use the render prop to customize the look and feel of the results.

function Main() {
  const [searchValue, setSearchValue] = useState("");

  return (
    <ReactiveBase {...configProps}>
      <ReactiveList
        componentId="SearchResult"
        dataField="Summary"
        size={12}
        pagination
        react={{ and: "SearchComponent" }}
        render={({ data }) => {
          return (
            <div className="mx-5 my-2">
                {data.map((item) => (
                  <div>
                    <h1>{item["Summary"]}</h1>
                    <p>{item["Text"]}</p>
                  </div>
                ))}
            </div>
          );
        }}
      />
    </ReactiveBase>
  );
}

Transforming the request:

Before we transform the request, we need to have several clarifications regarding when the network requests are performed and which network requests are performed.

• Suggestion query: When the user types into the SearchBox and doesn't hit enter, he sees a list of suggestions. Those suggestions are fetched by performing a network request whose type is suggestion. This query is debounced such that it fires only after an interval the user stops typing in the SearchBox and can be controlled by passing a numeric value to debounce prop.

• Search query: When the user hits Enter or selects a suggestion then a query of type search is fired. The search query is fired for both the components, SearchBox and ReactiveList. We can identify the component for which the query by looking at id property of the query. This id matches with the componentId of the specified component.

Now we are all set to transform network requests. Our search pipeline works if we provide a query of type search with a vectorDataField which we indexed earlier. We also need to provide a value which is the search query that would be vectorized and used for performing the KNN search. Rough structure of component queries is as below:

{
    "query": [
        {
            "id": "search",
            "type": "search",
            "dataField": [
                "Text",
                "Summary"
            ],
            "vectorDataField": "vector_data",
            "value": "good dog food",
            "excludeFields": [
                "vector_data"
            ]
        }
    ]
}

We can transform the network request by specifying a prop on ReactiveBase called transformRequest.

We would change only the query of type search and would augment it with vectorDataField and value of the search query. Here we would use the controlled value of the search component.

function Main() {
  const [searchValue, setSearchValue] = useState("");

  return (
    <ReactiveBase
      {...otherConfigProps}
      transformRequest={(req) => {
        const body = JSON.parse(req.body);
        // Transform query
        body.query = body.query.map((componentQuery) => {
          if (
            componentQuery.id === "SearchComponent" &&
            componentQuery.type === "search"
          ) {
            return { ...componentQuery, vectorDataField: "vector_data" };
          }
          if (
            componentQuery.id === "SearchResult" &&
            componentQuery.type === "search"
          ) {
            const searchQuery = body.query.find(
              (q) => q.id === "SearchComponent" && q.type === "search"
            );
            const searchValue = searchQuery.value;
            delete componentQuery.react;

            return {
              ...componentQuery,
              vectorDataField: "vector_data",
              value: searchValue,
            };
          }
          return componentQuery;
        });
        body.settings = {
          recordAnalytics: true,
          backend: "opensearch",
        };

        const newReq = { ...req, body: JSON.stringify(body) };
        return newReq;
      }}
    >
    </ReactiveBase>
  );
}

Finishing Up

Putting all of the above together and styling the app we would get a complete app below. We also add a few sample queries which can be used to perform search. Since the value of SearchBox is controlled, it is easy to make it possible. You can browse the codesandbox below. All the code is primarily present in src/App.jsx.

Summary

In this blog post, we discuss the implementation of K-nearest neighbors (KNN) search to improve search results, recommend items, and categorize data into clusters. We explain the concept of vector spaces and how they relate to KNN search. We then dive into the implementation of a search UI for a dataset of Amazon reviews, utilizing OpenAI's Vector Embeddings API, OpenSearch as the search engine backend, and ReactiveSearch for cloud hosting of OpenSearch and to build the UI for the app. We provide code examples and explanations for each step of the process, including indexing pipelines, search pipelines, and building the UI with ReactiveSearch components. Finally, we show how to transform the network request to complete the app.

Let's take a look at how the SearchBox component can be used in multiple ways to cover different use cases.

Voice Search

One interesting way to enhance the SearchBox is to enable voice search. You can do this by setting the showVoiceSearch prop to true. This adds a microphone icon to the search box, allowing users to search using their voice instead of typing. This can be especially useful for users on mobile devices or for those with accessibility needs.

    <SearchBox
        // ... other props
        showVoiceSearch={true}            
    />

Autosuggestion Search

Another powerful feature of the SearchBox is its ability to provide autosuggestions to users as they type. This can be accomplished using the autosuggest prop. When autosuggest prop is set to true, the SearchBox will query the index for suggestions based on the user's input. There are different types of autosuggestions you can enable using other props:

• Index Suggestions - Index suggestions are suggestions that show up from the indexed data in your search engine.

• Popular Suggestions - Popular suggestions are displayed from collecting the end-user search data based on popularity.

• Recent Suggestions - Recent suggestions are displayed from the current user's recent searches.

You can individually configure how each of these suggestions show when you enable autosuggestions.

Instant Search

One of the most powerful features is instant search, which allows search results to update in real time as the user types in the search box. This can be enabled by setting the autosuggest prop to false. With instant search, users can quickly see which search terms are yielding the most relevant results, and adjust their query as needed.

    <SearchBox
        // ... other props
        autosuggest={false}            
    />

Keyboard Shortcuts

Another useful feature is keyboard shortcuts, which can improve the speed and efficiency of the search process. The focusShortcuts prop allows you to define a list of keyboard shortcuts that will focus the search box when pressed.

For example, if you want to focus the search box when the user presses the / key, you can set focusShortcuts to ['/']. You can also define more complex keyboard shortcuts, such as SHIFT+A, by separating the key names with a +.

The string is case-insensitive and space in-between is optional. You can also pass several shortcuts ["SHIFT+A", "SHIFT+B"].

Instant Search with Pill based Suggestions

Combine Instant Search with Pill-based Suggestions to provide users with a highly interactive search experience. Instant Search offers real-time search results, while Pill-based Suggestions enable users to refine their search with predefined tags or filters displayed as pill-shaped buttons. This combination helps users easily navigate large datasets by providing a structured and intuitive way to explore data. It also reduces the likelihood of spelling errors or other mistakes, as users can quickly select the appropriate tag or filter to refine their search results.

The example here uses:

1. controlled usage for triggering the result query as per the user's wish/ needs. Click here to know more 🔗

2. render prop to render suggestions in a pill-based 💊 UI. Click here to know more 🔗

Tag-based Search UI

Create a highly customizable search experience with the Tag-based search UI, allowing users to select multiple search terms from a list of suggestions. Use the renderSelectedTags prop to define custom rendering functions for selected tags, offering users a more visually appealing interface or adding extra functionality to the tags.

<SearchBox
    componentId="searchSensor"
    // ... other props
    mode="tag"
/>

renderSelectedTags is used to custom render tags when the mode is set to tag.

Function param accepts an object with the following properties:

• values: Array<String> is an array of selected values

• handleClear: Function - (string) => void is a function to clear a tag value. It accepts the tag value(String) as a parameter

• handleClearAll: Function - () => void is a function to clear all selected values

    <SearchBox
        id="search-component"
        mode="tag"
        renderSelectedTags={({ 
            values = [], 
            handleClear, 
            handleClearAll }) => {
                // return custom-rendered tags 
            }
        }
    />

Featured Suggestions

Featured suggestions are a powerful feature in Reactive Search that can be used to display a set of custom search suggestions at the top of your search results. These suggestions are designed to be prominently displayed to your users and can be used to guide them toward popular or high-value search terms.

Some extensive use cases include:

• In-app navigation: Offer an in-app navigation experience through your search bar allowing users to navigate through your site or dashboard without needing to jump between multiple menus, or relying on header/footer navigations.

• Onboarding of new users: New users who are not familiar with the product or how to use it can just go to the search bar and search for the documentation of the product or search for help if needed. Through the in-app navigation, a user can search for frequently asked questions and navigate to the page effortlessly.

• User productivity with actions: When actions like toggling between the light mode or dark mode of your app can be done directly from the search bar, it can further enhance the search user experience.

To enable featured suggestions, you need to first set Featured suggestions via the ReactiveSearch dashboard, and then use the enableFeaturedSuggestions prop with searchboxId at query time.

Following is the dashboard workflow for setting the featured suggestions.

Following is the query time usage for returning featured suggestions.

<SearchBox
    // .. other props
    enableFeaturedSuggestions={true}
    searchboxId="your_created_searchbox_id"
    featuredSuggestionsConfig={{
        maxSuggestionsPerSection: 10,
        sectionsOrder: ['repositories', 'docs', 'functions'],
    }}
/>

Follow this guide to know more about Featured suggestions and how you can build one using the ReactiveSearch Control plane here 🚀.

AI Answer: Coming Soon ⏳

Get an AI Answer powered by ChatGPT, GPT-3 and GPT-4 models right in your SearchBox. Here's an example of how this works.

Summary

In summary, the SearchBox component from the @appbaseio/reactivesearch library enables you to create diverse, user-centric search experiences that cater to a wide range of use-cases. By leveraging its various customization options, you can provide your users with an intuitive, efficient, and engaging search experience that truly meets their needs.

Consider experimenting with different combinations of these features to find the perfect balance for your specific search scenario. As you refine your search experience, you'll be able to deliver an even more engaging experience to your end-users.

By employing user behavior tracking, personalized search delivers highly relevant results, boosting user satisfaction. If you're intrigued by the prospect of integrating personalized search into your website or search engine, you're in the right place.

Let's explore some popular applications of personalized search and how user preferences come into play:

1. Crafting algorithmic feeds based on browsing history: Platforms like YouTube and TikTok utilize personalized search to generate algorithmic feeds of videos that cater to users' interests. By analyzing watch history, likes, comments, shares, and other interactions, these platforms gain insight into users' preferences, serving up relevant and captivating content.

2. Curating feeds from user-selected topics: Personalized search also shines on news sites, Twitter, Quora, and other platforms where users input their interests or preferred topics. These platforms use this information to create feeds teeming with content that users find relevant and engaging. For instance, a news site might ask users to choose topics such as politics, sports, or entertainment, and then provide news articles on those subjects in their feed.

Some websites blend both approaches, using explicit user-selected topics along with tracking user history, to personalize search results and elevate the user experience. So go ahead, dive into the world of personalized search, and deliver content that resonates with your audience.

Here's the final personalized feed search UI that you will be building by going through this post:

The Building Blocks

To create a personalized search experience, we'll be using the ReactiveSearch platform and the open-source Apache 2.0 licensed ReactiveSearch UI kit for React.

• For this blog post's hosted UI, we have utilized a hosted ReactiveSearch service connected to an Elasticsearch backend. ReactiveSearch can also be deployed in self-host mode, as well as connect to other engines: OpenSearch, Solr, MongoDB and OpenAI are currently supported.

• To construct the search UI, we'll employ the React frontend UI components library from ReactiveSearch, which simplifies the development of search components such as search boxes, filters, and result displays. This streamlined approach paves the way for a truly customized search experience.

Building a search pipeline

Let's take a closer look at how traditional search functions. When a user searches for a specific term, like "football," the search engine retrieves documents from the search index. These documents are then ranked to determine the order in which they appear to the user. For our purposes, however, we want the search engine to "boost" or "promote" certain documents based on the end-users preferences. To achieve this, we'll introduce an additional step to promote relevant results before presenting them to the user.

In ReactiveSearch, this multi-step search query process, which adds context and more, is executed through pipelines. A pipeline consists of a sequence of search processing stages that query one or more search engines and return results to the user. We'll input the user's query into the search engine, which will collect and rank all pertinent documents. These documents will then advance to our "boost" step, where we'll enhance user-specific terms to ultimately deliver finely-tuned, personalized results.

In addition to the steps mentioned earlier, we can integrate a step in the pipeline to ensure that only authorized users can access the index. This authorization step can be positioned between the request processing and the search engine's ranking of documents. Pipelines offer incredible versatility, enabling you to incorporate features such as integrating results from ChatGPT, Google's Knowledge Graph, performing vector search, and more. To learn about ReactiveSearch pipelines and their capabilities, be sure to consult the documentation. You can learn about pipelines in the docs.

Authoring a pipeline using the dashboard

To begin, navigate to dash.reactivesearch.io and select "Pipelines" from the sidebar menu. Next, click on "Create a Pipeline." You'll find several pre-built templates catering to a wide range of use-cases; for our purposes, we'll opt for the "Basic Template." Upon making your selection, a text editor will appear, signaling the completion of this step.

The making of a pipeline

Upon opening the pipeline, you'll notice several components. Your pipeline should resemble the code snippet below. The first elements are enabled and description, both of which are self-explanatory.

Next, the routes property allows us to attach the pipeline to specific API endpoints, which a frontend can make a REST API call to retrieve results. We'll set the path within routes to /query-rules-boost.

Following this, we have envs, which are used to set the pipeline's operational context, including the index, domain settings, and more. We'll set the index to movies-demo-app here.

We would now see various parts of the pipeline. You would have a pipeline that would look something like below code snippet. We first have enabled and description. Their meanings are straightforward. enabled tells whether the pipeline is active and description provides some useful information about what the pipeline does.

Then, we have routes property which allows us to mount the pipeline on specific API endpoints which we can call to get the results. We would modify the path inside routes to /query-rules-boost.

After that, we have envs which describe the context in which the pipeline is running such as the index, domain settings etc. We would change the index to movies-demo-app.

We would also remove the script modify_request.js and it's corresponding reference.

{
    "enabled": true,
    "description": "Template to create a pipeline",
    "routes": [
        {
            "path": "/query-rules-boost",
            "method": "POST",
            "classify": {
                "category": "reactivesearch"
            }
        }
    ],
    "envs": {
        "index": [
            "movies-demo-app"
        ]
    },
    "stages": [
        {
            "id": "auth",
            "use": "authorization"
        },
        {
            "id": "query",
            "use": "reactivesearchQuery",
            "continueOnError": false
        },
        {
            "id": "es_query",
            "use": "elasticsearchQuery",
            "continueOnError": false
        }
    ]
}

Stages are the building blocks of the pipeline, representing the steps of data transformation, as illustrated in the diagram above. These stages are executed sequentially, constructing the results that are ultimately presented to the user. The "auth" stage verifies whether a user has permission to access the index or perform the intended action. Next, the reactivesearch query stage converts the ReactiveSearch query into an Elasticsearch query. Finally, the es_query stage executes a query on the Elasticsearch index, collecting the results that are then delivered to the user.

Incorporating a stage to refine search results

Following the diagram, we can introduce an additional stage after retrieving results from the Elasticsearch index to enhance specific search outcomes. We can incorporate a stage called boost, you can see its reference in the documentation.

{
    // Same pipeline options as above with additional "boost" stage
    "stages": [
        {
            "id": "auth",
            "use": "authorization"
        },
        {
            "id": "query",
            "use": "reactivesearchQuery",
            "continueOnError": false
        },
        {
            "id": "es_query",
            "use": "elasticsearchQuery",
            "continueOnError": false
        },
        /*
        Score documents containing 
        "Harry Potter" better
        */
        {
            "id": "boost-harry-potter",
            "use": "boost",
            "continueOnError": false,
            "inputs": {
                // The field to query in documents
                "dataField": "original_title",
                "value": [
                    "Harry Potter"
                ],
                "boostType": "score",
                "boostMaxDocs": 3 // Boost only top 3 docs
            }
        }
    ]
}

We're setting the stage to boost up to 3 documents that contain the value Harry Potter in the original_title index field. We start with a very specific value to test out the personalization in action.

The dashboard provides a realtime validation testing view where we can test the above pipeline to confirm that we indeed get the top 3 documents of the hits containing Harry Potter in their original_title field.

Building a search UI

With our pipeline configured, it's time to witness it in action. We'll use the React flavor of the ReactiveSearch library to develop the UI. However, you can use the ReactiveSearch dashboard's no-code UI builder, or Vue.JS, vanilla JS, React Native or Flutter libraries to do this. See the choosing UI library section.

npx and go!

We will use create-react-app to get the boilerplate code. Execute the following command in the terminal to create a React app named "feed-search":

npx create-react-app feed-search
cd feed-search
yarn add @appbaseio/reactivesearch
code .   # Fire up your code editor

@appbaseio/reactivesearch is ReactiveSearch's React npm package.

As you open the code editor, notice the src/App.js file, we will be editing this file primarily to build the search UI.

Connecting to the pipeline

Let's connect the pipeline using the ReactiveBase component. Each Reactivesearch UI has a ReactiveBase component as the connector component, where you configure the index and cluster URL. Here, we do this by specifying the endpoint property.

// src/App.js
import "./App.css";
import {
  ReactiveBase
} from "@appbaseio/reactivesearch";

function App() {
  return (
    <>
      <ReactiveBase
        endpoint={{
          url: "https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io/query-rules-boost",
          method: "POST",
        }}
        reactivesearchAPIConfig={{
          recordAnalytics: true,
        }}
      >
      </ReactiveBase>
    </>
  );
}

export default App;

Now we can add some more components like a SearchBox which shows a search bar where a user inputs their search query. We can also add a ReactiveList to display the results returned from the pipeline. You can find more information in the docs for SearchBox and ReactiveList.

import "./App.css";
import {
  ReactiveBase,
  ReactiveList,
  ResultCard,
  SearchBox,
} from "@appbaseio/reactivesearch";

function App() {
  return (
    <>
      <ReactiveBase
        endpoint={{
          url: "https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io/query-rules-boost-static",
          method: "POST",
        }}
        reactivesearchAPIConfig={{
          recordAnalytics: false,
        }}
      >
        <div className="row mt-4 p-3">
          <SearchBox
            dataField={["original_title"]}
            componentId="BookSensor"
            highlight
            URLParams
            size={5}
            enablePredictiveSuggestions
            showClear
            renderNoSuggestion={() => "No suggestions found."}
          />
          <ReactiveList
            componentId="SearchResult"
            dataField="original_title"
            size={12}
            className="result-list-container"
            pagination
            react={{
              and: "BookSensor",
            }}
            render={({ data }) => {
              // Render results returned from the pipeline
            }}
          />
        </div>
      </ReactiveBase>
    </>
  );
}

export default App;

You can look at the final UI below. Observe that we boost 3 "Harry Potter" documents when the search query is empty.

Getting user preferences

We aim to promote specific search results dynamically based on an individual's preferences. This can be achieved by tracking browser history and boosting search results based on recent searches. In this case, we'll allow users to explicitly select the topics they're interested in using a straightforward tag-based selection. Users can choose up to three movie genres, which the search engine will then prioritize in the search results.

To accomplish this, we will create a component inside a file called TagSelector.js. This will feature a select input, enabling users to choose their preferred genres. Additionally, we'll need to incorporate the selected options into our request payload. This data will then be utilized within the pipeline to enhance search results. We can achieve this by using the transformRequest prop of ReactiveBase, which modifies the search request before making a call to the pipeline API.

import "./App.css";
import {
  ReactiveBase,
  ReactiveList,
  ResultCard,
  SearchBox,
  SelectedFilters,
} from "@appbaseio/reactivesearch";
import Navbar from "./Navbar";
import { useEffect, useState } from "react";
import TagSelector from "./TagSelector";

import styles from "./App.module.css";

function App() {
  const [options, setOptions] = useState(["Comedy"]);
  const [isLoading, setIsLoading] = useState(false);

  useEffect(() => {
    if (options) {
      setIsLoading(true);
      setTimeout(() => {
        setIsLoading(false);
      }, 1000);
    }
  }, [options]);

  return (
    <>
      <Navbar />
      <div className="p-3">
        <div className="row">
          <div className={styles.userPreference}>User Preference:</div>
          <div className="col">
            <TagSelector options={options} setOptions={setOptions} />
          </div>
        </div>
      </div>
      {isLoading ? (
        <div className="row mt-4 p-3">Resetting topics...</div>
      ) : (
        <ReactiveBase
          {...samePropsAsBefore}
          transformRequest={(req) => {
            const body = JSON.parse(req.body);
            body.customData = options;
            const newReq = { ...req, body: JSON.stringify(body) };
            return newReq;
          }}
        >
          {/*
              Search and other UI components
          */}
        </ReactiveBase>
      )}
    </>
  );
}

export default App;

If you try out the live search UI, the request body payload should look like below:

POST /query-rules-boost
{
    customData: ["Comedy", "Horror"] // user selected options
}

Incorporating Preferences within the Pipeline

Accessing Context

With data from the frontend now structured within the request body, we need to access it inside the pipeline using context.

Each pipeline operates within a specific context. This context can be accessed within the stages to dynamically modify their input. First, however, we must parse the request and incorporate the selected options into the context. To achieve this, we'll create a script called promoteParams.js, which will be added as a stage prior to our boost stage. This script introduces an additionalParams property to the context object, which can be accessed within the stage as {{additionalParams}}.

// promoteParams.js
function handleRequest() {
    const requestBody = JSON.parse(context.request.body);
    const customData = (requestBody && requestBody.customData) || [];

    return {
        additionalParams: {
            paramOne: customData[0] || '',
            paramTwo: customData[1] || '',
            paramThree: customData[2] || '',
        },
    };
}

You can add the script as a stage using the dashboard. First, reference the script inside the pipeline as a stage and then add a file named promoteParams.js with the above content. Add the script before the boost stages.

{
    // Same pipeline options as above with additional script stage
    "stages": [
        //Other stages same as above
        // Add below stage before the boost stage
        {
            "id": "addPromoteParams",
            "scriptRef": "promoteParams.js"
        },
        {
            "id": "boost-harry-potter",
            "use": "boost",
            "continueOnError": false,
            "inputs": {
                // The field to query in documents
                "dataField": "original_title",
                "value": [
                    "Harry Potter"
                ],
                "boostType": "score",
                "boostMaxDocs": 3 // Boost only top 3 docs
            }
        }
    ]
}

Using Context:

Each pipeline has a context in which it's executing. We can access this context as an input to a stage.

Now we can reference the context inside the boost stages. We also add two more boost stages to handle all three genres.

{
    // Same pipeline options as above with dynamic input from context
    "stages": [
        // Other stages
        {
            "id": "addPromoteParams",
            "scriptRef": "promoteParams.js"
        },
        {
            "id": "boostOne",
            "use": "boost",
            "continueOnError": false,
            "inputs": {
                "dataField": "genres",
                "value": [
                    "{{additionalParams.paramOne}}"
                ],
                "boostType": "score",
                "boostMaxDocs": 3
            }
        },
        {
            "id": "boostTwo",
            "use": "boost",
            "continueOnError": false,
            "inputs": {
                "dataField": "genres",
                "value": [
                    "{{additionalParams.paramTwo}}"
                ],
                "boostType": "score",
                "boostMaxDocs": 3
            }
        },
        {
            "id": "boostThree",
            "use": "boost",
            "continueOnError": false,
            "inputs": {
                "dataField": "genres",
                "value": [
                    "{{additionalParams.paramThree}}"
                ],
                "boostType": "score",
                "boostMaxDocs": 3
            }
        }
    ]
}

Congratulations, we've now updated the pipeline to use the user preferences passed from the frontend search UI. The complete frontend code and live hosting of the search UI is shown below:

Summary

In this post, we explored the concept of personalized search and its implementation using ReactiveSearch and Elasticsearch. We learned how to create a pipeline with multiple stages to process and enhance search results based on user preferences. Additionally, we built a search UI using the ReactiveSearch library and a React app, allowing users to explicitly define their interests through a tag-based selection. We also delved into accessing and modifying context within the pipeline stages to dynamically adjust search results. By combining these techniques, we're able to provide users with a customized and engaging search experience.

ReactiveSearch Pipelines offer a V8 engine-based JavaScript runtime (similar to CloudFlare workers) that you can run alongside search engines to craft relevant search experiences for SaaS, Enterprise and E-Commerce use cases.

Additional links to explore more:

https://reactivesearch.io/

https://github.com/awesome-reactivesearch/feed-search-ui

https://docs.reactivesearch.io/docs/pipelines/how-to/

When building your search experience with client-side JavaScript, you may worry that search engines can’t crawl or render your URLs and that this may hurt your ranking. While some search engines are getting better at processing sites with client-side JavaScript search, you can also do a lot to optimize your website for search engines without sacrificing the user experience.

Your search result pages are directly accessible through a URL

As users interact with your search interface, the URL should dynamically update and reflect the refinements they’ve selected and their actions.

This allows them to share links to your website, enhancing the overall usability of your search and encouraging backlinking.

Try opening the code sandbox in a new tab and see how the URLs are changing with every significant search interaction.

The approach clearly improves the user experience, but there are concerns or well-spread myths about dynamic URLs that require busting.

#1 Myth: "Dynamic URLs cannot be crawled."

#2 Myth: "Dynamic URLs are okay if you use fewer than three parameters."

Having read the above references we can safely conclude that Dynamic URLs don't necessarily hurt your SEO and should be favored.

Your widgets use crawlable a tags with href attributes

Search Engines can follow your links only if they use proper <a> tags with resolvable URLs.

• ✅ Dos:

  • <a href="https://example.com">

  • <a href="/relative/path/file">

Note that links are also crawlable when you use JavaScript to insert them into a page dynamically as long as it uses the markup shown above.

don'ts:

• <a routerLink="some/path">

• <span href="https://example.com">

• <a onclick="goto('https://example.com')">

This is relevant when you have opted to use custom markup.

Your URLs are readable

A site's URL structure should be as simple as possible. Consider organizing your content so that URLs are constructed logically and in a manner that is most intelligible to humans.

For every ReactiveSearch component/ widget, the basic routing configuration injects every search refinement into the URL as a query string parameter. These parameters are inferred from the global search state.

https://fwmwo.csb.app/?BookName=Paradise

// The below information is saved legibly in the URL params above
// componentId = 'BookName'
// component's value = 'Paradise'

// you can visit the above mentioned app to see how the URls are handled out of the box

✅ Dos

• Simple, descriptive words in the URL

  https://en.wikipedia.org/wiki/Aviation

• Localized words in the URL, if applicable.

  https://www.example.com/lebensmittel/pfefferminz

• Use UTF-8 encoding as necessary. For example, the following example uses UTF-8 encoding for Arabic characters in the URL:

  https://www.example.com/%D9%86%D8%B9%D9%86%D8%A7%D8%B9/%D8%A8%D9%82%D8%A7%D9%84%D8%A9

🚫 Don'ts

• Using non-ASCII characters in the URL:

  https://www.example.com/نعناع , https://www.example.com/杂货/薄荷, etc.

• Unreadable, long ID numbers in the URL:

  https://www.example.com/index.php?id_sezione=360&sid=3a5ebc944f41daa6f849f730f1

Although ReactiveSearch by default uses query strings for the Search UIs, but also provides support for customization to switch to path based routing.

ReactiveBase supports two props to allow users to control how the URL of the search UI behaves.

• getSearchParams

Enables you to customize the evaluation of query-params-string from the URL (or) any other source. If this function is not set, the library will use window.location.search as the search query-params-string for parsing selected values. This can come in handy if the URL is using hash values.

• setSearchParams

Enables you to customize the setting of the query params string in the URL by providing the updated query-params-string as the function parameter. If this function is not set, the library will set the window.history via pushState method.

In our example attached below,

We leverage setSearchParams to receive a URL decorated with query params and manipulate the query params to generate a human-friendly route, finally applying it using window.history.pushState method.

setSearchParams = (urlWithQueryParams) => {
  // build URL with category and sub-category path
  window.history.pushState({ path: urlWithPath }, '', urlWithPath);
}

On the other hand getSearchParams callback is used to parse the path-based URL and turn it into library supported URL with query params.

getSearchParams = () => {
  // extract query params from the path
  // path $host/Best%20%Buy/Mobile?color=red&page=2
  // Extract query params: category=Best%2-Buy&sub-category=Mobile&color=red&page=2
  return newURLwithAllParams
}

Voila 🎊 Try selecting the category and the subcategory in the example to see the route's path changing.

Resources

• Keep a simple URL structure

• The Complete Guide on Structuring Your URLs Correctly

Your URLs reflect the structure of your website

Some search engines use your URL structure to infer the architecture of your website, understand the context of a page, and enhance its relevance to a particular search query.

Well-structured URLs offer users a quick hint about the page topic and how the page fits within the website.

For example, if a website has categories and sub-categories, each category should be reachable through https://mywebsite.com/<category>/ and each sub-category through https://mywebsite.com/<category>/<sub-category>.

✅ Dos

• https://mywebsite.com/Car-Equipment/

• https://mywebsite.com/Women-Clothing/T-Shirts/

🚫 Don'ts

• https://mywebsite.com/search?category=Cars-Equipement

• https://mywebsite.com/search?categorylvl1=Clothing&categoryLvl2=T-shirts

Resources -

• Better presentation of URLs in search results

Your site is using a pre-rendering technique

The first question is, "Why do we even need to pre-render an HTML page"?

In the CSR technique, which has come into our lives with modern browsers, websites send very little HTML response to connection requests. This HTML response contains JavaScript codes that make up the content of the page. Unless these JavaScript codes are executed, the website is a blank page. The browser renders these JavaScript files, creates the content, and presents the web page to us.

🚫✋🏻 But wait ✋🏻🚫, The server sends little to no HTML but a script to be executed on the client side which in turn is responsible for generating the HTML. The crawlers have got a problem with this. The crawlers primarily like HTML and don't wait to execute JS to parse the content of the page.

Not all search engine crawlers can process JavaScript successfully or immediately. Fortunately, there are many ways around it.

Server-side rendering (SSR)

This technique fetches your data and renders a JavaScript website on the server before sending it to the browser. This process is commonly implemented through modern frameworks such as React, Angular, and Vue.

Let's help you out 🤝

Reactivesearch internally runs on a redux store. With Server Side Rendering, you can handle the initial render when a user (or search engine crawler) first requests your app. To achieve the relevant results on an initial render, we need to pre-populate the redux store of ReactiveSearch.

Visit our docs here to know more.

A quick demo -

Resources

• Understand the JavaScript SEO basics

• Javascript SEO: What is SSR/CSR? Advantages and Disadvantages

• How You Render Can Affect Your SEO (CSR vs SSR vs Dynamic Rendering)

• 🚀 Server side rendering with ReactiveSearch

You can download it for your search project with the following command:

npm install @appbaseio/reactivesearch@latest

In this blog post, I will share a summary of the key enhancements and changes included in the v4 release.

• Support for React 18 😎

  ReactiveSearch and ReactiveMaps are fully compatible with React 18.x and above with the 4.x releases.

• Removal of search engine DSL from the library

  To prevent a DoS scenario and as a security best practice, we're removing frontend DSL generation. ReactiveSearch library will instead use Reactivesearch API to declare the search intent. This will require the use of a ReactiveSearch API server and you can choose one of the two path ways to do this:

  • As a ReactiveSearch (previously appbase.io) customer, you already have the ReactiveSearch API server hosted in the cloud. There are no usage changes as compared to ReactiveSearch v3's usage in this scenario.

  • As an open-source user, you have the choice to run ReactiveSearch API server as an open-source docker container. The API server should be configured to point to a search engine: Elasticsearch, OpenSearch and Solr are currently supported.

• Removal of DataSearch & CategorySearch components

  Besides SearchBox component, v3.x exported two components for auto-suggestions: DataSearch & CategorySearch. 4.x removes these additional components, as the SearchBox component is versatile and allows creating a search UI with many possible options.

• Richer Analytics

  With v4.x, we are enabling a richer analytics experience for our users, thanks to the use of analytics.js by ReactiveSearch. Users can save searches, mark searches as favorites, tag custom events, in addition to recording search and clicks. Check out the docs to learn more

• New components

  • TreeList

    

    TreeList creates a selection-based hierarchical tree UI component that is connected to your search index's category > sub-category > sub-sub-category type of hierarchical fields. It supports both single and multiple item selections.

  • Chart Components

    

    • ReactiveSearch now has chart components to build reporting UIs with advanced search capabilities, or to integrate a reporting view into your search UI.

      • Pie Chart ↗️

      • Bar Chart ↗️

      • Histogram Chart ↗️

      • Line Chart ↗️

      • Scatter Chart ↗️

      • Reactive Chart ↗️

        Got a custom chart to render? ReactiveChart is a generic component to render any chart UI supported by the Apache E-charts library. Additionally, it supports pre-built charts (pie, bar, line, histogram and scatter) to cover the most common chart use cases that can be configured using declarative props.

• Removal of Deprecated props

  We have also removed the following deprecated props and instead recommend using their alternatives

• Simplified SSR API

With the release of v4, the SSR implementation is more robust and easier, requiring very little config from users. 😌

Here's a SSR usage example with the new API:

import React, { Component } from 'react';
import {
    // ... other components
    getServerState // the method to calculate initial state
} from '@appbaseio/reactivesearch';

const components = {
    // component props-settings
};

export default class Main extends Component {
    static async getInitialProps({ pathname, query }) {
        return {
            // 1st Argument
            // pass the reference to the app, i.e. Main in this case
            // The new method saves you from explicitly specifying the
            // component type, props, and source
            // 2nd Argument
            // The queryString or the query object to consider 
            // URLParams while computing the initialState at server side
            store: await getServerState(Main, query); 
        };
    }

    // In ReactiveBase (refer below)
    // the contextCollector is another prop required for RS SSR
    // to do some magic internally
    // you should pass it without worrying about its source
    // we handle it for you

    render() {
        return (
            <>
                <ReactiveBase 
                    {...components.settings}                     
                    initialState={props.initialState}                   
                    contextCollector={props.contextCollector}
                >                    
                    {/* ... other components ...*/}
                </ReactiveBase>
            </>
        );
    }
}

The new usage mechanism takes care of a lot of configuration from the user on its own now, significantly reducing human error while writing the config.

See the new SSR usage in action with the CodeSandbox below:

💫 Additionally, you can read the docs here to know more about how ReactiveSearch provides SSR support out of the box.

Summary

We went over React v18 support, introduction of new components for TreeList and Charts, richer analytics support, low-config SSR usage support, and removal of frontend DSL generation from the library.

Take ReactiveSearch v4 for a spin and let us know your feedback on the GitHub issues.

You can follow the quickstart guide over here to build a search UI with ReactiveSearch v4.

ReactiveSearch now supports 5 built-in chart components: Pie Chart, Bar Chart, Histogram, Line, and Scatter, and comes with a generic component - ReactiveChart which can use any of the Apache E-charts UI components. You can read more about creating charts with ReactiveSearch over here.

Who is this for?

The need for the overall representation of data in a single glance is an essential requirement for all businesses. That’s where a charts UI comes into play.

A charts UI can be used to:

• Enable your business team to visualize the important business metrics in a single glance,

• Share performance and ROI reports with your clients,

• Empower your customers and end-users to visualize metrics relevant to them,

• Collate existing dashboards into a unifying dashboard.

There are dashboard builder tools such as Kibana and Grafana which let you create Elasticsearch-powered dashboards with no code. We recommend them whenever they fit your use case. However, they become limiting as soon as you need to programmatically filter and visualize the data, e.g. filter data for the logged-in user and set access controls on the data that’s visible, need to integrate other JavaScript code within your dashboard, or extend further by calling other backend routes.

What are we building?

We are going to build a drill-down charts dashboard app that has the following features:

• An explore page where we can apply various filters and see the number of results it finds.

• A search page where we can see the details of the results.

• The explore page can drill down to the search page with all the facets selected on the page.

You can take a look at the live version below.

What do we need?

We would be using some tools to help make this complex task simple.

1. Dataset: To make a great UI we should have a good dataset. We would be using an e-commerce dataset that has a vast amount of product data across categories. We would also need an index to store the data.

   You can set up and install an Elasticsearch server by following the official installation guide, or you can create a free account at reactivesearch.io which provides Elasticsearch hosting as a service and is easy to use. For simplicity, we will be using reactivesearch.io service to get started.

   I’ve already created an index with the dataset. You can check out the dataset from above over here in the data browser tool Dejavu, which is built by reactivesearch.io.

2. ReactiveSearch — A declarative, props-driven UI library for querying and managing the search state and comes with over 30 UI component presets. This lets you customize both your queries and UIs to render in minutes and comes with extensive documentation.

3. Antd: A UI components library so that we don't have to focus on making our UI beautiful and consistent.

4. React Router: We are going to build a multi-page application, which would need to handle URL navigation. We would use react-router-dom for that.

Let's get started

Initial Setup

We can bootstrap a react project by running yarn create react-app [app-name] or npx create-react-app [app-name]. This would create a project directory and initialize a package.json file in the root with all the necessary packages and scripts.

We would also need to install react-router-dom and antd library. This app was made using v4 of both libraries.

yarn add react-router-dom@4 antd@4

You would also need to install @appbaseio/reactivesearch which would help us create the search UI.

yarn add @appbaseio/reactivesearch

Below, shows the app after running the create react app command.

Building the Explore page

This is the main page of our app. This would contain different chart facets which we would use to narrow down our search results. We would be using @appbaseio/reactivesearch library which provides a range of facets.

Firstly, we need to modify our App.js file inside to configure our routing like below. We would create two pages, Explore.js and Search.js, inside a new directory. We would name the directory pages. Below routing configuration shows the Explore page on /explore , Search page on /search and for any other route, it would redirect to Explore page.

// src/App.js
import { BrowserRouter as Router, Switch, Route, Redirect } from 'react-router-dom';
import Explore from './pages/Explore';
import Search from './pages/Search';

function App() {
    return (
        <Router>
            <Switch>
                <Route exact path="/explore">
                    <Explore />
                </Route>
                <Route path="/search">
                    <Search />
                </Route>
                <Route path="/">
                    <Redirect to="/explore" />
                </Route>
            </Switch>
        </Router>
    );
}

export default App;

We would start building the Explore.js page now. First, we need to connect to the Elasticsearch index storing our dataset. For that, we would use ReactiveBase component from the library. The index name is best-buy-dataset and we have configured the credentials and URL. Since this is made using reactivesearch.io we are also passing enableAppbase to be true.

// Explore.js
import { ReactiveBase } from "@appbaseio/reactivesearch";
import React from "react";

function Explore() {
  return (
    <ReactiveBase
      app="best-buy-dataset"
      url="https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io"
      enableAppbase
    >
    {/* Our App goes here */}
    </ReactiveBase>
  );
}

Then we are going to add a chart facet. For this, we would use ReactiveChart component and pass it required props to show us a pie chart. componentId differentiates this facet from others. dataField determines the field in your dataset/index on which you want to query. Finally, chartType is the chart you want to render. It can be pie, bar, line, etc. You can look at how we can configure different chart facets in the documentation.

// src/pages/Explore.js
import { ReactiveChart, ReactiveBase } from "@appbaseio/reactivesearch";

function Explore() {
  return (
    <ReactiveBase
      app="best-buy-dataset"
      url="https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io"
      enableAppbase
    >
      <ReactiveChart
        componentId="Category"
        dataField="class.keyword"
        chartType="pie"
      />
    </ReactiveBase>
  );
}

We can use the Card component from antd to wrap each chart facet which would make the UI look a little polished.

// src/pages/Explore.js
import { ReactiveChart, ReactiveBase } from "@appbaseio/reactivesearch";

function Explore() {
  return (
    <ReactiveBase
      app="best-buy-dataset"
      url="https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io"
      enableAppbase
    >
      <Card>
          <ReactiveChart
            componentId="Category"
            dataField="class.keyword"
            chartType="pie"
          />
      </Card>
    </ReactiveBase>
  );
}

You can see the complete Explore page. We would render an empty page for the Search page. We have also added some styles in src/App.css and imported them in src/App.js. You can check the source code in the codesandbox below.

Building the Search page

Now Search.js would be different from the Explore page, such that it would show the results with all the facets by the side. It would look like something below.

We would add facets on the side of the results. Firstly, create a new file under src/components and name it Facets.js. The facets we would configure are SingleList, BarChart, DynamicRangeSlider and LineChart. The final configuration should look like below.

// src/components/Facets.js
import {
  DynamicRangeSlider,
  ReactiveChart,
  SingleList,
} from "@appbaseio/reactivesearch";

export default function Facets() {
  return (
    <>
      <SingleList
        componentId="Category"
        dataField="class.keyword"
        URLParams
        loader="Loading..."
      />
      <ReactiveChart
        componentId="SubCategory"
        dataField="subclass.keyword"
        chartType="bar"
        type="term"
        URLParams
        useAsFilter
        loader="Loading..."
      />
      <DynamicRangeSlider
        componentId="ReviewAverage"
        dataField="customerReviewAverage"
        range={{ start: 0, end: 5 }}
        rangeLabels={(min, max) => ({
          start: min + " ⭐️",
          end: max + " ⭐️",
        })}
        loader="Loading..."
        showHistogram
        URLParams
      />
      <ReactiveChart
        componentId="Color"
        dataField="color.keyword"
        chartType="line"
        type="term"
        URLParams
        useAsFilter
        loader="Loading..."
      />
    </>
  );
}

// src/pages/Search.js
import { ReactiveBase, ReactiveList } from '@appbaseio/reactivesearch';
import React from 'react';
import Facets from '../components/Facets';

export default function Search() {

    return (
        <ReactiveBase
            app="best-buy-dataset"
            url="https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io"
            enableAppbase
            initialQueriesSyncTime={1000}
        >
             <Facets />
        </ReactiveBase>
    );
}

We also have to show results on the right side. For that, we are going to use another component called ReactiveList. Not only does it show results but it also updates itself when one of the facets changes. It also gives us a nice interface to render data in our dataset.

We would add a ReactiveList by configuring it's componentId, dataField, and renderItem method. We've already seen what componentId and dataField do. As for the renderItem method, it is a method that passes down each individual document from the Elasticsearch index which you can use to customize the look and feel. Apart from that we also configure an additional property called react. This tells what facet components would update the results in this list. We should pass the componentId of the facets when doing this. You can read more about configuring ReactiveList component in the documentation. We also use Col and Row layout components from antd which would help us to create responsive layouts easily.

import { ReactiveBase, ReactiveList } from "@appbaseio/reactivesearch";
import { Col, Row } from "antd";
import React from "react";
import Facets from "../components/Facets";

export default function Search() {
  return (
    <ReactiveBase
      app="best-buy-dataset"
      url="https://a03a1cb71321:75b6603d-9456-4a5a-af6b-a487b309eb61@appbase-demo-ansible-abxiydt-arc.searchbase.io"
      enableAppbase
      initialQueriesSyncTime={1000}
    >
      <Row>
        <Col xs={24} md={8} className="p10">
          <Facets />
        </Col>
        <Col xs={24} md={16} className="p10">
          <ReactiveList
            componentId="SearchResult"
            dataField="original_title"
            className="result-list-container"
            from={0}
            size={5}
            renderItem={(data) => {
              return (
                <div className="resultItem" key={data._id}>
                  <img
                    src={data.image}
                    alt="Book Cover"
                    className="resultItem__img"
                  />
                  <div className="resultItem__body">
                    <div className="book-header">{data.name}</div>
                    <div className="flex column justify-space-between">
                      <div>
                        <div>
                          <span className="resultItem__category">
                            {data.class}
                          </span>{" "}
                          <span>></span>{" "}
                          <span className="resultItem__subCategory">
                            {data.subclass}
                          </span>
                        </div>
                        <div className="ratings-list flex align-center">
                          Sale price: <span className="">{data.salePrice}</span>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              );
            }}
            react={{
              and: [
                "Category",
                "SubCategory",
                "Color",
                "ReviewAverage",
                "SearchBox",
              ],
            }}
            includeFields={[
              "class",
              "subclass",
              "name",
              "image",
              "salePrice",
              "categoryPath",
            ]}
          />
        </Col>
      </Row>
    </ReactiveBase>
  );
}

We would also add a SearchBox from @appbaseio/reactivesearch. You can check the look and feel of the page with all the components below.

Drill down behavior

We have built both pages. Now we need to connect them. We would add a card at the bottom, which would show the number of results filtered by charts, clicking which would lead to the search page.

For that, we would also use a ReactiveList component. We would use the render method to customize our render. It is different from renderItem such that it gets all the documents at once. It also gets some additional props useful to display the result stats such as the total number of documents. We would also hide the default result stats by passing showResultStats as false and hide pagination as well.

We would also add an onClick method which would take us to the search page. The onclick handler would apply all the filters from the Explore page to the Search page.

Since the above functionality relies on URL params to pass this information, we would need to pass URLParams prop as true in all the facet components on both pages.

// src/pages/Explore.js
// ... import statements

function Explore({ history }) {
    return (
        <ReactiveBase
            {...configuration}
        >
        {/* Chart facets */}    
        <Row>
                <ReactiveList
                    componentId="ListComponent"
                    dataField="albumTitle"
                    pagination={false}
                    infiniteScroll={false}
                    showResultStats={false}
                    renderNoResults={() => null}
                    react={{ and: ['Category', 'SubCategory', 'ReviewAverage', 'Color'] }}
                    className="fullWidth"
                    render={({ data, ...props }) => {
                        return (
                            <Card
                                className="fullWidth"
                                onClick={() => {
                                    const urlLocation = new URL(window.location.href);
                                    const urlSearchParams = new URLSearchParams(urlLocation.search);
                                    history.push(`/search?${urlSearchParams}`);
                                }}
                            >
                                <h1 className="resultsCard">
                                    {!props.loading
                                        ? `${props.resultStats.numberOfResults} matched the above criteria. View now.`
                                        : 'View Search Results'}
                                </h1>
                            </Card>
                        );
                    }}
                />
            </Row>
        </ReactiveBase>
    );
}

Congratulations! We've created a super-duper charts dashboard with a search interface. You can look at the complete working app below.

Summary

I hope you enjoyed this tutorial. We introduced so many things in this post, which might get overwhelming. So, let's summarize what you need to keep in mind while building such an app.

• Step 1: Set up a react project. (We used CRA for convenience)

• Step 2: Add an explore page where we can apply facets and see the number of results filtered.

• Step 3: We created a search page to show the results with the applied facets.

• Step 4: We connected them with a drill-down behavior.