R and Solr Integration Using Solr's REST APIs

Solr is the most popular, fast and reliable open source enterprise search platform from the Apache Luene project.  Among many other features, we love its powerful full-text search, hit highlighting, faceted search, and near real-time indexing.  Solr powers the search and navigation features of many of the world's largest internet sites.  Solr, written in Java, uses the Lucene Java search library for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it easy to use from virtually any programming language including R.  

We invested significant amount of time integrating our R-based data-management platform with Solr using HTTP/JSON based REST interface.  This integration allowed us to index millions of data-sets in solr in real-time as these data-sets get processed by R.  It took us few days to stabilize and optimize this approach and we are very proud to share this approach and source code with you.  The full source code can be found and downloaded from datadolph.in's git repository. 

The script has R functions for:

• querying Solr and returning matching docs
• posting a document to solr  (taking a list and converting it to JSON before posting it)
• deleting all indexes, deleting indexes for a certain document type and for a certain category within document type

     # query a field for the text and return docs

      querySolr <- function(queryText, queryfield="all") {

        response <- fromJSON(getURL(paste(getQueryURL(), queryfield, ":", queryText, sep="")))

        if(!response$responseHeader$status) #if 0

          return(response$response$docs)

      }

      # delete all indexes from solr server

      deleteAllIndexes <-function() {

        response <- postForm(getUpdateURL(),

                             .opts = list(postfields = '{"delete": {"query":"*:*"}}',

                                          httpheader = c('Content-Type' = 'application/json', 

                                                         Accept = 'application/json')

                                          ssl.verifypeer=FALSE

                             )

        ) #end of PostForm

        return(fromJSON(response)$responseHeader[1])

      }

      # delete all indexes for a document type from solr server 

      # in this example : type = sports

      deleteSportsIndexes <-function() {

        response <- postForm(getUpdateURL(),

                             .opts = list(postfields = '{"delete": {"query":"type:sports"}}',

                                          httpheader = c('Content-Type' = 'application/json', 

                                                         Accept = 'application/json'),

                                          ssl.verifypeer=FALSE

                             )

        ) #end of PostForm

        return(fromJSON(response)$responseHeader[1])

      }

      # delete indexes for all baskeball category in sports type from solr server 

      # in this example : type = sports and category: basketball

      deleteSportsIndexesForCat <-function(category) {

        response <- postForm(getUpdateURL(),

                             .opts = list(postfields = 

                               paste('{"delete": {"query":"type:sports AND category:', category, '"}}', sep=""),

                                          httpheader = c('Content-Type' = 'application/json', 

                                                         Accept = 'application/json'),

                                          ssl.verifypeer=FALSE

                             )

        ) #end of PostForm

        return(fromJSON(response)$responseHeader[1])

      }

      #deletePadIndexesForCat("baskeball")

      #Post a new document to Solr

      postDoc <- function(doc) { 

        solr_update_url <- getUpdateURL()

        jsonst <- toJSON(list(doc))

        response <- postForm(solr_update_url,

                             .opts = list(postfields = jsonst,

                                          httpheader = c('Content-Type' = 'application/json', 

                                                         Accept = 'application/json'),

                                          ssl.verifypeer=FALSE

                             )) #end of PostForm

        return(fromJSON(response)$responseHeader[1])

        ########## Commit - only if it doesn't work the other way ###############

        #return(fromJSON(getURL(getCommitURL())))

      }

Happy Coding!