EDABench is a 120-item evaluation benchmark built specifically for this domain, covering five task categories with zero training data contamination. All four systems were judged in a single Claude Sonnet batch using identical prompts — ensuring the scores are directly comparable with no calibration drift between runs.

This table is the core evidence of the project. We tested every one of the 120 benchmark questions against four different systems and scored each answer on four dimensions: is it factually correct, is it complete, is it actionable (can an engineer use it to fix the problem), and is it specific enough to be useful? The overall "Answer Quality" score is the minimum of these four — meaning one weak dimension drags the whole score down. This is deliberately strict: a response that is factually perfect but vague scores low, because vague answers don't help engineers.

The headline finding is clear: the full GraphRAG system (knowledge graph + vector search + reranker + Claude) scores 0.482, which is 6.7 times higher than a bare AI model at 0.072. But the more nuanced story is in the rows between. The fine-tuned LoRA model scores 0.431 without any retrieval at all — it learned EDA patterns from training data — but it can't access the knowledge graph, so its graph hit rate is 0%. Vector-only search scores just 0.202, proving that plain document retrieval isn't enough for this domain. The knowledge graph provides the structured context that makes the difference.

These two charts break down performance by the type of question being asked. The left chart shows that the system performs best on constraint generation and RTL questions (0.517 and 0.511) — these are areas where the knowledge graph has dense, well-connected data about tool configurations and code patterns. DRC Rule Lookup and Cross-Tool Knowledge score lower (0.422 each), but for different reasons: DRC questions involve PDK rule IDs that the entity extractor can't yet match, while cross-tool questions require connections across multiple tool boundaries.

The right chart tells an even clearer story. Graph hit rate — the percentage of answers that actually used information from the knowledge graph — is 89% for RTL and Cross-Tool questions, 83% for Constraint Generation, and 71% for Error Diagnosis. DRC Rule Lookup sits at 0% because the entity extractor doesn't recognize PDK rule naming patterns like METAL1.S.1. This is a known limitation with a straightforward fix (regex pattern extension), not a fundamental system weakness. The high graph hit rates everywhere else confirm that the knowledge graph is actively contributing to answers, not just sitting unused.

These two case studies are the most concrete evidence in the project because they come from real bugs discovered during actual chip design experiments. ED-005 is a crash that only happens with a specific version of OpenROAD (build 26Q1-2900-gdf79404cd8) when running hierarchical global routing on the ibex processor design. The full GraphRAG system scores 2.5× higher than any other approach because the knowledge graph contains the exact build hash, the crash stage, and the version relationship — information that plain text search can't piece together from scattered log files and changelogs.

ED-002 flips the story. Here, the fine-tuned LoRA model wins decisively (0.70 vs 0.20) because the WNS timing sign flip between ORFS v3.0 and 26Q1 was well-represented in the training data. The model learned this specific pattern and can reproduce it without needing to search anything. This is the key insight about complementarity: structured retrieval excels when the answer depends on following version-specific connections through a graph, while fine-tuning excels when the answer matches a pattern the model has seen during training. Neither approach alone covers all cases — the full system needs both.

The difficulty chart reveals a surprising pattern: expert-level questions actually score higher (0.460) than hard-level ones (0.390). This isn't a flaw — it reflects how knowledge graphs work. Expert questions tend to involve specific, named tools and version numbers that the graph can match precisely. Hard questions often involve ambiguous multi-factor causation where the right answer depends on interactions between several components, making both retrieval and synthesis more difficult. Easy questions score highest (0.597) as expected, confirming the system handles straightforward lookups well.

The cost breakdown tells the project's financial story. Synthetic Q&A generation consumed 87% of the budget ($208.64) — this was the single largest investment because generating 13,000 high-quality training pairs required extensive AI processing. Everything else was remarkably cheap: the knowledge graph extraction cost nothing extra (85% was done with free regex and NER tools), LoRA fine-tuning on a single GPU cost $9.09, and the full ORFS parameter sweep on AWS Spot instances cost just $2.41. The three cost-saving strategies listed — Batch API discounts, tiered processing, and Spot instance discipline — are what made a $240 total possible against a $2,685 original estimate.

The system was designed as a six-stage pipeline where each component feeds the next — and each was independently validated before moving forward. We started by collecting every publicly available document about open-source chip design tools: manuals, error logs, GitHub issues, forum posts, and release notes — over 14,000 files totaling 252 million tokens. After removing duplicates, we used AI to generate 13,000 synthetic question-and-answer pairs that teach the system how EDA engineers actually ask questions and what good answers look like.

From that cleaned corpus, we built two retrieval systems that work in parallel. The knowledge graph (Neo4j) maps structured relationships — which tool versions introduced which bugs, which fixes apply to which errors, which design rules conflict with which configurations. The vector index (Weaviate) enables fast similarity search over text passages. When a query arrives, both systems are searched simultaneously, results are merged and re-ranked by a cross-encoder model, and the best evidence is handed to an AI model (Claude) that writes a clear, cited answer. A separately fine-tuned Mistral-7B model adds domain vocabulary and response formatting. The entire pipeline was built incrementally over 16 weeks, with every decision documented in a 1,414-line experimental trail.

No research system ships without gaps, and being transparent about them is as important as reporting what works. Four areas need attention in upcoming work. First, the entity extractor currently cannot match PDK design rule IDs (like METAL1.S.1), which is why DRC Rule Lookup shows 0% graph hit rate despite reasonable answer quality — a straightforward regex pattern extension will fix this. Second, only 33 of the 120 EDABench items have full KG-grounded expected nodes annotated; the remaining 87 are scored on answer quality alone, which limits how precisely we can measure retrieval accuracy. Third, the swerv_wrapper design was excluded from the ORFS parameter sweep because it requires a 64 GB instance, meaning 3 of the 12 planned experimental runs were deferred. Fourth, the ASAP7 7nm academic PDK rules are not yet in the corpus, so queries about ASAP7 design rules return low-confidence answers until that documentation is added to the ingestion pipeline. None of these limitations affect the core findings — the 6.7× improvement and 67.5% graph hit rate are measured on the components that work — but addressing them will strengthen both the benchmark and the system for future use.