Sample, refine, repeat
There’s no question that many of the early case assessment tools (“ECA”) currently available can significantly reduce the amount of data that needs to be reviewed. Advanced searching, filtering and categorization are valuable techniques in the pursuit of effective data targeting, but the process wrapped around them is equally as important.
Is the way you use case assessment tools defensible?
The ad hoc nature of an ECA tool may appear effective, but often the tools don’t adequately track and measure search and validation efforts. If tools are being used to select what will or won’t be reviewed (as opposed to assessing contents of a corpus), then the specific use of the tool — and the process used to apply it — become much more relevant. Without a formal, measured process that includes automated functionality plus human validation (accompanied by an understanding of how a particular technology works) you may be putting yourself at risk. The process should include sampling and testing of search criteria in an iterative fashion looking at what hit and did not hit on the criteria.
Sedona refers to the combination of tools and human input as “The Intelligent Use of Tools”:
…merely adopting sophisticated automated search tools, alone, will not necessarily lead to successful results. Lawyers must recognize that, just as important as utilizing the automated tools, is tuning the process in and by which a legal team uses such tools, including a close involvement of lead counsel. This may require an iterative process which importantly utilizes feedback and learning as tools, and allows for measurement of results.
Judge Grimm in the Victor Stanley case states that “common sense” dictates that sampling of ESI search results is required to validate the criteria:
…Common sense suggests that even a properly designed and executed keyword search may prove to be over-inclusive or under-inclusive … [t]he only prudent way to test the reliability of the keyword search is to perform some appropriate sampling of the documents … in order to arrive at a comfort level that the categories are neither over-inclusive nor under-inclusive.
All in all, the best approach is to maximize the benefit of automated ECA tools by applying industry recognized best practices to the process. Ad hoc searches are useful, but a more measured, formalized search process that involves a closer analysis of the data and results is much more appropriate.
Filed under: 2009 Conference
