Ensuring RFP Parsing Accuracy: Testing Requirement Extractors
Start with a Known Solicitation
Select an RFP we've already meticulously parsed by hand. This baseline allows us to compare the tool's extractions against a known set of requirements. Choose an RFP with a mix of straightforward and complex requirements for a robust test case.
Identify Key Sections
Focus on critical sections like Section L (Subcontracting Plan) and Section M (Evaluation Factors). These sections often contain the most detailed and specific requirements. For example, Section L may include FAR clause 52.219-9, which mandates a small business subcontracting plan. Section M might detail evaluation factors such as technical merit and past performance.
Extract Requirements Manually
Document every requirement, noting the page number and section for reference. This manual extraction serves as the gold standard against which we'll measure the tool's performance. For instance, if the RFP mentions NAICS code 541611 (Administrative Management and General Management Consulting Services), note the exact page and context.
Run the Solicitation Through the Extractor
Process the RFP using the requirement extraction tool. Ensure the tool runs without interruptions and captures all sections of the solicitation. This step is crucial for generating the data we'll use in our accuracy assessment.
Compare Extractions to Manual Parse
Measure the tool's accuracy by comparing its output to our manual extraction. Create a comparison table that lists each requirement, the tool's extraction, and the manual extraction. This will help us identify discrepancies quickly.
Calculate Misses
Count how many requirements the tool failed to identify. Misses are critical because they represent gaps in the tool's capability to capture all necessary information. For example, if the tool misses a requirement related to FAR clause 52.222-40 (Service Contract Labor Standards), this could lead to non-compliance.
Identify False Positives
Note any extractions that don't align with the actual requirements. False positives can mislead proposal teams and lead to wasted effort. For instance, if the tool incorrectly identifies a requirement related to IDIQ (Indefinite Delivery, Indefinite Quantity) contracts when the RFP discusses BPAs (Blanket Purchase Agreements), this needs to be flagged.
Assess Mis-Classifications
Check for instances where the tool incorrectly categorized a requirement. Mis-classifications can distort the understanding of the solicitation's priorities. For example, if a technical requirement is misclassified under past performance, this could skew the proposal strategy.
Spot-Check Against Source
Verify a sample of extractions by cross-referencing the source page numbers. This step ensures that the tool's extractions are not only present but also accurate in context. For instance, if the tool extracts a requirement from page 15, verify that the context on page 15 matches the extraction.
Analyze FAR Clause Extraction
Evaluate the tool's performance on specific FAR clauses like 52.219-9 (Small Business Subcontracting Plan). Accurate extraction of these clauses is vital for compliance and proposal success. If the tool fails to extract or misclassifies these clauses, it undermines its reliability.
Review NAICS Code Accuracy
Assess the tool's ability to correctly identify and classify NAICS codes relevant to the solicitation. For example, if the RFP specifies NAICS code 334112 (Electronic Computer Manufacturing), ensure the tool accurately extracts and classifies this code. Incorrect NAICS codes can lead to misaligned proposals.
Next Steps: Validate Before You Rely
Before trusting an extractor with live RFPs, validate its accuracy using our protocol. VETR offers tools to facilitate this process, ensuring you only rely on verified, accurate extractions. If you want a structured way to score your current capture pipeline, the free VETR readiness assessment will walk you through it in under five minutes. For more detailed insights, explore our agency-specific playbooks or NAICS-code playbooks.
To get started with VETR's comprehensive features, register for a free trial today. If you have any questions or need further assistance, contact our sales team for personalized guidance.