Notes

OTP DSL Generator Agent — Complete Project Overview
1. What Is This Project?
The DSL Generator Agent is a Python-based automated pipeline that generates Domain-Specific Language (DSL) code for 52 banking process patterns (credit card processing, clearing, rewards, account transfers, etc.) used in Fiserv's OTP (Optis Transformation Platform). Instead of hand-writing thousands of DSL files, this agent reads an Enterprise Knowledge Graph (EKG) stored in a GraphDB instance, builds an Intermediate Representation (IR), and generates 4 layers of DSL files that compile into Java via the NexFlow compiler and then build with Maven.

2. The Pipeline (End-to-End Flow)
Step-by-step:

SPARQL Queries (Q1–Q20) hit the EKG to fetch process definitions, steps, schemas, fields, Kafka topics, MCC context, COBOL code, SHACL shapes, etc.
An IR Builder assembles all EKG data into Python dataclasses (ProcessDefinition, ProcessStep, ServiceDomainSchema, etc.)
Four DSL generators (L1–L4) emit .proc, .schema, .xform, and .rules files.
Each pattern folder gets a nexflow.toml config file. NexFlow compiles DSLs → Java. Maven builds the final artifacts.
3. The Four DSL Layers
Layer Extension Grammar File What It Defines
L1 — Process .proc ProcDSL.g4 Process orchestration: steps, stream receive/emit, joins, enrichments, branches, MCC context lookups
L2 — Schema .schema SchemaDSL.g4 Data schemas: fields with types, types blocks (BIM objects from SHACL), optional sections (entries, migration, parameters, state, computed, pii)
L3 — Transform .xform TransformDSL.g4 Data transformations: tojsonobject + getvaluebypath mappings, type conversions (to_string, to_int, etc.), optional validate_input/validate_output/onError blocks
L4 — Rules .rules RulesDSL.g4 Business rules: decision tables (from COBOL EVALUATE), procedural rules (from COBOL IF/MOVE/COMPUTE), PCF retrievals, Zeta annotations
The ANTLR4 grammar files (.g4) are the authoritative source of truth for what constitutes valid DSL syntax.

4. EKG Data Sources & SPARQL Queries
EKG Instance: otp_ekg_qa repository on GraphDB
Credentials: fiserv_read_write_qa / Fiserv@int

Query Purpose Key Namespace
Q1 List all processes ps: (process)
Q2 Process steps for a process ps:
Q3 Step details (SD, SO, predicates) ps:, sd:
Q4 Service domain schemas sd:
Q5 Field mappings (COBOL→JSON) cobol: (16,880 fields)
Q5b Data dictionary by entity cobol:entityCode
Q8 BIAN meta-model — (0 results in QA)
Q10 MCC context items mcc: (185 items)
Q11 Kafka topic registry kr:Topic (31 topics)
Q12 MCC context for process mcc:
Q12b MCC enriched with SD/SO labels mcc: + rdfs:label
Q13 PCF fields for FC name optis:associatedPCF
Q14 Paragraph IRIs for FC optis:usedIn
Q15 COBOL source code per paragraph cbl:code
Q16/Q17 Zeta call sequences/rules — (data gap, returns 0)
Q18 SHACL shapes for SD shacl:
Q19 SHACL properties shacl:
Q20 BIM compositions —
Key Namespace Discoveries:

cobol: (http://fiserv.com/ont/cobol#) — 16,880 field names, 4,063 JSON mappings
kr:Topic (http://fiserv.com/ont/kafka-registry#) — 31 Kafka topics, maps SD+SO → topic names
mcc:Context (http://fiserv.com/ont/mcc#) — 185 context items, defines what data each process needs at start/runtime
shacl: — Shape-based field definitions with XSD types, used for SHACL-enriched schemas
optis:pcfFieldsAffected — 174 PCF field entries
5. Key Architectural Components
dsl_compiler/
├── config.py # EKG endpoint, credentials, topic_prefix
├── compiler_pipeline.py # Main orchestrator — fetches, builds IR, generates DSLs
├── ekg/
│ ├── sparql_queries.py # All Q1–Q20 SPARQL queries
│ ├── ekg_fetcher.py # HTTP methods for all queries
│ ├── kafka_registry.py # KafkaRegistry class — topic lookup by SD+SO
│ ├── mcc_registry.py # MccRegistry class — context items per process
│ └── kafka-mapping.json # Local Kafka topic mapping file
├── ir/
│ ├── ir_builder.py # Builds IR from raw EKG data + registries
│ └── process_ir.py # Dataclasses: ProcessDefinition, ProcessStep, ServiceDomainSchema, ShaclProperty, BimChildEntity
├── generators/
│ ├── l1_proc_generator.py # L1 Process DSL
│ ├── l2_schema_generator.py # L2 Schema DSL (SHACL+BIM types)
│ ├── l3_xform_generator.py # L3 Transform DSL (getvaluebypath)
│ └── l4_rules_generator.py # L4 Rules DSL (COBOL-backed)
├── cobol/
│ └── cobol_block_parser.py # Deterministic COBOL parser (IF, EVALUATE, MOVE, COMPUTE)
└── rules/
├── fc_context.py # FCContextBuilder — fetches FC→paragraph→COBOL chain
└── rules_mapper.py # RulesMapper — COBOL blocks → DSL decision tables & rules
6. How the IR Builder Works
The IR Builder is the heart of the pipeline. For each process:

Entity Resolution (4-strategy cascade):

Strategy 0: KafkaRegistry by SD+SO (highest priority, exact topic names)
Strategy 1: EKG Q8 BIAN meta-model (returns 0 in QA env)
Strategy 2: Q3 SO response analysis
Strategy 3: SD prefix strip (fallback)
Hardcoded fallback if all fail
Schema Building: Creates ServiceDomainSchema objects with:

cobol: fields filtered by cobol:entityCode
SHACL properties from Q18-Q19 (types with XSD→DSL mapping)
BIM child entities from Q20
MCC Context: Attaches contextRequiredAtStart and contextAddedAtRuntime from mcc:Context

FC Context: For steps with ps:implements (FunctionalCapability), fetches PCF fields → paragraph IRIs → COBOL source code

7. Rules Generation (L4) — The COBOL Pipeline
FC Name (ps:implements) → PCF Fields (Q13) → Paragraph IRIs (Q14) → COBOL Source (Q15) → Parse → DSL Rules
The most complex part. For each process step that implements a Functional Capability (FC):

COBOL Block Parser: Reads column-based COBOL (cols 1-6 sequence, col 7 indicator, cols 8-72 code)
Block types parsed: IF/ELSE/END-IF, EVALUATE/WHEN, MOVE, COMPUTE, GO TO, PERFORM
EVALUATE blocks → decision_table (with header columns + data rows + separator)
IF/MOVE/COMPUTE blocks → procedural rules
PCF references: BSF-* fields → retrievePCFValue("field", "Type") calls
Zeta annotations: Business objectives from Q17 prepended to rule descriptions
8. Output Structure (Per Pattern)
Each of the 52 patterns generates a folder like:
Generated_DSLs/otp-ai-dsl-clearing/
├── nexflow.toml # NexFlow build config
└── src/
├── proc/
│ └── clearing.proc # L1 process orchestration
├── schemas/
│ ├── cr_credit_card_facility.schema # L2 schemas (367 total)
│ ├── bq_account_insight.schema
│ └── ...
├── transforms/
│ ├── update_credit_card_facility.xform # L3 transforms (229 total)
│ └── ...
└── rule_defs/ # Changed from rules/ to avoid keyword collision
├── cash_item_fee.rules # L4 rules (205 total)
└── ...

Output totals across all 52 patterns:

52 .proc files (one per process)
367 .schema files (multiple schemas per process)
229 .xform files (one per schema that needs transformation)
205 .rules files (generated from COBOL/FC context)
52 nexflow.toml files (build configuration)
9. Key Design Decisions & Fixes Made Over 8 Sessions
Session Date Key Work
1 March 18 Initial pipeline: EKG connection, SPARQL queries, basic L1-L3 generation
2 March 19 Discovered kr:Topic namespace, added Kafka registry, live topic lookup
3 March 20 Discovered mcc:Context namespace, MCC registry, context lookups in L1
4 March 20 Found cobol: namespace (16,880 fields!), fixed Q5/Q5b, entity-code filtering, join blocks
5-6 March 23-24 L4 Rules pipeline: COBOL parser, FC context builder, rules mapper, decision tables
7 March 31 Build fixes: rule_defs/ path (keyword collision), empty-block guards, MCC schema stubs, Zeta annotations, PCF retrievePCFValue()
8 April 2 SHACL enrichment: Q18-Q20, BIM types in schemas, getvaluebypath transforms, cr_/bq_ prefix fix
9 April 4 QA validation analysis, grammar-based verification, nexflow.toml generation
10. Critical Build Fixes Applied
rules/ → rule_defs/: ProcDSL.g4 has rules as a reserved keyword in importPathSegment. All import paths and L4 output paths changed to rule_defs/.
Empty block guards: COBOL blocks that parse to empty then/else get set _noop = true placeholder.
Expression sanitization: Dangling arithmetic operators (+, -, *, / at line end) are stripped.
string → text: RulesDSL grammar uses text not string as field type.
COBOL literals: SPACE/SPACES → "", ZERO/ZEROS/ZEROES → 0.
// call → apply: COBOL PERFORM statements map to apply keyword.
MCC schema stubs: _ensure_mcc_schemas() creates schema entries for MCC context items resolved to Kafka topics.
11. QA Validation Analysis Results
A QA team ran a validation framework against both AI-generated and hand-written DSLs:

AI average score: 67.90 | Hand-written average score: 60.38 (AI scores HIGHER)
AI blockers reported: 216 | Hand-written blockers: 997 (AI has FEWER)
Grammar-based analysis revealed:

~1,212 of the reported issues are NOT actual bugs — they're either grammar-correct constructs, optional features, style checks, or QA validator limitations
Only 2 real bugs found:
Process stream naming (91 blockers across 22 patterns): When the same EKG step repeats, the _N dedup suffix is applied to the consumer from reference but the producer into never emits with that suffix → stream name mismatch
Decision table column mismatches (~22 data rows): COBOL mapper sometimes generates data rows with different column counts than the header row
12. The 52 Process Patterns
The system generates DSLs for all these banking processes:

Monetary: Clearing, Credit Allocation, Balance Aging/Reaging, Adjustment Processing, Authorization Hold, Authorization Matching, Transaction Settlement, Cycle Processing, Daily Processing, Daily Bank Totals, Daily Posted Monetary Transaction Details, Day Zero Monetary Balances, Delinquency, Processing Financial Adjustment Request

Non-Monetary: Account Transfer, Auto Non-Mon, CapMaster, Day Zero Non-Monetary, New Account, Non-Mon Persist, Non-Mon with PFA, CSPA

Rewards: Rewards Daily Calculation, Rewards Distribution, Rewards Monthly Calculation, Rewards Positions Synchronization, Rewards Transaction Processing

Synchronization (Day-0 & Day-N): Account Data, ACS Data, Auth Log Data, CHM Data (Day0), Client Data (Day0/DayN), CSPA Data (Day0/DayN), DMM Data (Day0/DayN), Mon Data, Non-Mon Data, PCF Data (Day0/DayN), Rewards Data, Rules Data (Day0/DayN), ValTran Mon Data

Other: Behavior Scoring, Bontran (Fraud & Risk), DMM Day-0 Data Load, PCF Daily Delta Load, Plastic Reissue, Rules Day-0 Load, Scheduled Event Report

13. How To Run The Pipeline
# Activate virtual environment
.venvScriptsActivate.ps1

# Generate DSLs for a single process
python main.py --process "Clearing"

# Generate DSLs for ALL 52 processes
python main.py --all

# Build with NexFlow + Maven (after DSL generation)
run_all_builds.bat
14. Remaining Work
Fix #1 — Stream naming bug: In l1_proc_generator.py, when dedup suffix _N is added to a step name, the into (produce) block must also emit the suffixed stream name so downstream consumers can find it.
Fix #2 — Decision table columns: In rules_mapper.py, ensure column counts in data rows always match the header row in decision tables.
Full Maven build validation: Run run_all_builds.bat end-to-end to verify NexFlow compile + Maven build passes for all 52 patterns.
15. Key Files To Know
File What It Does
compiler_pipeline.py Start here — main orchestrator
ekg/sparql_queries.py All 20 SPARQL queries
ekg/ekg_fetcher.py HTTP calls to GraphDB
ir/ir_builder.py Builds IR from EKG data
ir/process_ir.py All dataclass models
generators/l1_proc_generator.py Process DSL generator
generators/l2_schema_generator.py Schema DSL generator (SHACL+BIM)
generators/l3_xform_generator.py Transform DSL generator
generators/l4_rules_generator.py Rules DSL generator (COBOL-backed)
cobol/cobol_block_parser.py COBOL source parser
rules/fc_context.py FC→COBOL chain builder
rules/rules_mapper.py COBOL→DSL rules mapper
ekg/kafka_registry.py Kafka topic resolution
ekg/mcc_registry.py MCC context items
This covers the entire project end-to-end: architecture, data sources, all 20 SPARQL queries, all 4 DSL layers with their grammars, the COBOL rules pipeline, all 52 patterns, all 8 sessions of development work, the QA validation results, the 2 remaining bugs, and how to run everything.