Learn What You Need to Know About CAL (Continuous Active Learning) & Automation
If you are doing a regular linear review, in reality, your relevant documents are likely spread out among hundreds, thousands, or even hundreds of thousands of documents. You could cull documents before review with searches but you could miss relevant documents or get a lot of false positives. Review is a very time-consuming process and it is hard to achieve consistency in between reviewers. We do leverage a traditional Venio Assisted Review process (VAR) and have invested in the development of CAL (Continuous Active Learning) over the last year.
Understanding Venio Assisted Review (VAR)
With VAR, the reviewers or subject matter experts (SME) would review seed documents and then based on those seed documents, the system would manually create a model to predict unreviewed documents in order to complete the review faster. However, this is an iterative process and the SME may need to review documents in many rounds. The SME (Subject Matter Expert) will need to invest a lot of time and Assisted Review expertise is required to create and test that the model is efficient in predicting documents. Once the model is deemed accurate enough, then the reviewers can start looking at relevant documents. One large drawback of VAR is that it is more expensive and time-consuming.
Understanding CAL vs. VAR
|
Training Documents |
Training Model (Engine) |
Review |
|
CAL is coded by reviewers |
CAL is continuously updated based on a Reviewer’s coding decisions |
In CAL, Review and Training are the same |
|
CAR is coded by the SME only |
VAR is based on Reviewer’s and SMEs decisions on seed documents |
In VAR, Review starts after Training is done |
In CAL, the main differences are that while the Reviewers are reviewing, it is also actively learning and updating models. It is also re-sorting documents that Reviewers needs to review so that the next set of documents the reviewer is given for review is rich with documents relevant to the scope of review (based on continuously updated model), whereas in VAR, once the training and prediction phases are over and reviewers start to review the predicted documents, the model remain static, only seeds coded by SME during the training phases are used for model. In CAL, the review and training happen in parallel, saving time. In addition, review can be done by others besides the SME. Similarly, in CAL, a reviewer spends more time reviewing relevant documents as compared to VAR.
Leveraging CAL (Continuous Active Learning) on the VenioOne Platform
On the VenioOne platform, there are a few key steps – (1) a CAL profile is created with documents to be reviewed in the review set, then (2) the reviewers begin to review documents until the defined thresholds are hit. It is during this process that the review and training happen in parallel.
One of those thresholds is Review Relevance Threshold. This is relevant recall target for the case. For example, if your threshold is 80%, then a reviewer only needs to review enough documents to recall 80% of “projected” or “calculated” relevant documents in the case. While the reviewer reviews, (3) it is also training the model. (4) Reranking is done every time a batch of documents is checked-in and is then added to the model.
In addition, the other key threshold you can set is the batch richness. Batch Richness is the ratio of relevant documents found by reviewers to the total documents in a batch. So a reviewer can stop reviewing once the “Review Relevance Threshold” is met and if the batch richness falls below the threshold.
Reducing Time and Increasing Efficiency and Effectiveness
In this particular example, by the time the reviewer reviewed 1805 documents, they had met both thresholds set for the review-set and they had found most of the relevant documents out of 19,524 utilizing CAL.
The Venio Systems team proudly invites you to join us for an exclusive preview of our Continuous Active Learning (CAL) technology. Our powerful automation and analytics technology makes reviewing data, big and small, effortless and quick. Learn about our commitment to better discovery and better decisions. Please register here for our upcoming webinars.
Sincerely,
