DevStory #17: Performant low code streamlit based full text search unlocking data from 900 million pages / 150 million Texas land records
Documents are filed with the county courts registering transactions related to deeds, notes, settlements, and leases. These are public records. Historical information as far back as the 1920s until the present day is stored in the form of tiffs and is not easily accessible as only a few organizations have access to this information as they have collected this information over the years.
Attorneys, paralegals, title researchers, run sheet generating landmen at oil and gas land departments, banks and trusts, mortgage brokers, real estate brokers, and surveyors research these tiff images. These resources are used in real estate, oil and gas, title research, mortgage lending, and investment industries.
Records and information are from predominantly non-urban settings such as small metropolitan areas, micropolitan counties, and rural counties where natural resources are significant economic drivers.
This is an information goldmine, but not accessible in a user-friendly manner. To make searching this data easy and help stakeholders research and nail down relevant information, it was decided to extract the text in this information using OCR, then enhance and extract datasets from the text, index the text and build a search capability akin to Google Search so as to identify documents of interest easily, research the documents and then purchase the documents through the website increase revenue. All this on a shoestring budget.
The data from 150 million documents and a total of 900 million pages were extracted as detailed in DevStory #16: AI and Full text search on information extracted from 900 million land record pages.
A search index resident on SearchLight needs to be queried to extract data in a performant manner. The following movie details the functionality implemented against the full text index on Searchlight.
The search criterion is Named Entities in the document. People who are grantors or grantees. There is one instance of a partial address used to find a document amid all the documents. The land records span all of Texas. These tools and techniques can be applied to large repositories of scanned documents simplifying hidden information retrieval opening up a goldmine of information and opportunity.
The backend process to generate the process has been detailed here. A graphic depicting the process to extract information from TIFF images and index the underlying documents that enables this easy search is as follows.
